[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:29 dist-test-slave-4dmq krb5kdc[14475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:29 dist-test-slave-4dmq krb5kdc[14475](info): set up 2 sockets
Sep 25 05:05:29 dist-test-slave-4dmq krb5kdc[14475](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 05:05:30.614339 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:30 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:30.724086 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:34541
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:30.800920 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:30.801160 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:30.801198 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:30.802402 14488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:30.802465 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:30.802491 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:30.802523 14488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:30.802544 14488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:30.803980 14488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:30.804242 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:30.804455 14488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:05:30.809585 14488 server_base.cc:1034] running on GCE node
W20260925 05:05:30.809548 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:30.809566 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:30.809559 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:30.810026 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:30.810324 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:30.811488 14488 hybrid_clock.cc:648] HybridClock initialized: now 1790312730811485 us; error 35 us; skew 500 ppm
Sep 25 05:05:30 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:30.813889 14488 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:30.814342 14488 webserver.cc:460] Webserver started at http://127.14.33.62:33081/ using document root <none> and password file <none>
I20260925 05:05:30.814564 14488 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:30.814630 14488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:30.814775 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:30.815752 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "36bf318f5c074c898bb48d395f7494b4"
format_stamp: "Formatted at 2026-09-25 05:05:30 on dist-test-slave-4dmq"
server_key: "86f8abd31e20d772e4fabfba49b7b0bb"
server_key_iv: "740fd5d4e6ccb11b62e28d28c1ea7626"
server_key_version: "encryptionkey@0"
I20260925 05:05:30.816102 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "36bf318f5c074c898bb48d395f7494b4"
format_stamp: "Formatted at 2026-09-25 05:05:30 on dist-test-slave-4dmq"
server_key: "86f8abd31e20d772e4fabfba49b7b0bb"
server_key_iv: "740fd5d4e6ccb11b62e28d28c1ea7626"
server_key_version: "encryptionkey@0"
I20260925 05:05:30.817407 14488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:30.818159 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:30.818318 14488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:30.818413 14488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "36bf318f5c074c898bb48d395f7494b4"
format_stamp: "Formatted at 2026-09-25 05:05:30 on dist-test-slave-4dmq"
server_key: "86f8abd31e20d772e4fabfba49b7b0bb"
server_key_iv: "740fd5d4e6ccb11b62e28d28c1ea7626"
server_key_version: "encryptionkey@0"
I20260925 05:05:30.818485 14488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:30.842767 14488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:30.844973 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:30.845165 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:30.849176 14488 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:34541
I20260925 05:05:30.849251 14555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:34541 every 8 connection(s)
I20260925 05:05:30.849579 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:30.850364 14556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:30.852470 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4: Bootstrap starting.
I20260925 05:05:30.853039 14556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:30.853334 14556 log.cc:824] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:30.853965 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4: No bootstrap required, opened a new log
I20260925 05:05:30.855473 14556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf318f5c074c898bb48d395f7494b4" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34541 } }
I20260925 05:05:30.855602 14556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:30.855630 14556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bf318f5c074c898bb48d395f7494b4, State: Initialized, Role: FOLLOWER
I20260925 05:05:30.855787 14556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf318f5c074c898bb48d395f7494b4" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34541 } }
I20260925 05:05:30.855849 14556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:30.855870 14556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:30.855917 14556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:30.856415 14556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf318f5c074c898bb48d395f7494b4" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34541 } }
I20260925 05:05:30.856520 14556 leader_election.cc:302] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36bf318f5c074c898bb48d395f7494b4; no voters:
I20260925 05:05:30.856690 14556 leader_election.cc:288] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:30.856849 14559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:30.856926 14556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:30.857019 14559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [term 1 LEADER]: Becoming Leader. State: Replica: 36bf318f5c074c898bb48d395f7494b4, State: Running, Role: LEADER
I20260925 05:05:30.857137 14559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf318f5c074c898bb48d395f7494b4" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34541 } }
I20260925 05:05:30.857504 14559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36bf318f5c074c898bb48d395f7494b4. Latest consensus state: current_term: 1 leader_uuid: "36bf318f5c074c898bb48d395f7494b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf318f5c074c898bb48d395f7494b4" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34541 } } }
I20260925 05:05:30.857580 14559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:30.857658 14560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36bf318f5c074c898bb48d395f7494b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bf318f5c074c898bb48d395f7494b4" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34541 } } }
I20260925 05:05:30.857751 14560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:30.858155 14568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:30.858489 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 14488
I20260925 05:05:30.858603 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:30.858700 14568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:30.859368 14468 external_mini_cluster.cc:1428] Setting key acd281f9340afd58ced09590639d9a91
I20260925 05:05:30.861493 14568 catalog_manager.cc:1347] Generated new cluster ID: 883c71f891da49609440316ab447d436
I20260925 05:05:30.861544 14568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:05:30 dist-test-slave-4dmq krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:30.877000 14568 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:30.877517 14568 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:30.880581 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:30.860153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.14.33.62:34541)
0925 05:05:30.860336 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:30.860345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:30.860601 (+ 256us) server_negotiation.cc:407] Connection header received
0925 05:05:30.866370 (+ 5769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:30.866378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:30.866432 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:30.866577 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:30.868357 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:30.869913 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:30.871344 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:30.871945 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:30.872056 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:30.876071 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:30.876075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:30.876075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:30.876086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:30.878587 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:30.879067 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:30.879070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:30.879070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:30.879177 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:30.879434 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:30.879435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:30.879436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:30.879695 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:30.879870 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:30.880113 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 05:05:30.880250 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260925 05:05:30.886809 14568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36bf318f5c074c898bb48d395f7494b4: Generated new TSK 0
I20260925 05:05:30.886960 14568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:30.919895 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34541
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:30.995757 14582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:30.995945 14582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:30.995967 14582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:30.997001 14582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:30.997035 14582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:30.997061 14582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:30.998373 14582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:30.998533 14582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:30.998683 14582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:31.003115 14590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.003144 14587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.003119 14588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:31.003408 14582 server_base.cc:1034] running on GCE node
I20260925 05:05:31.003603 14582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:31.003827 14582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:31.004959 14582 hybrid_clock.cc:648] HybridClock initialized: now 1790312731004960 us; error 39 us; skew 500 ppm
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:31.007584 14582 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:31.007954 14582 webserver.cc:460] Webserver started at http://127.14.33.1:38107/ using document root <none> and password file <none>
I20260925 05:05:31.008137 14582 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:31.008186 14582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:31.008277 14582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:31.009346 14582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "4f1262b1bc3f4d3eaaf849f9c3df3bdf"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "1c927706304962a70421a8746d3ebf36"
server_key_iv: "925a70bb090b9057a308daf8b9cbf6bd"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.009658 14582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "4f1262b1bc3f4d3eaaf849f9c3df3bdf"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "1c927706304962a70421a8746d3ebf36"
server_key_iv: "925a70bb090b9057a308daf8b9cbf6bd"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.010910 14582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:31.011565 14597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.011734 14582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:31.011807 14582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "4f1262b1bc3f4d3eaaf849f9c3df3bdf"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "1c927706304962a70421a8746d3ebf36"
server_key_iv: "925a70bb090b9057a308daf8b9cbf6bd"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.011855 14582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:31.031596 14582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:31.033078 14582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:31.033231 14582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:31.033383 14582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:31.033587 14582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:31.033613 14582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.033631 14582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:31.033643 14582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.038796 14582 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:35607
I20260925 05:05:31.038863 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:35607 every 8 connection(s)
I20260925 05:05:31.039115 14582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:31.044239 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 14582
I20260925 05:05:31.044333 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:31.044482 14468 external_mini_cluster.cc:1428] Setting key 36b85d2c1a63488d2e0b825e4714951c
I20260925 05:05:31.047870 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.039731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:34157 (local address 127.14.33.62:34541)
0925 05:05:31.039771 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.039773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.039926 (+ 153us) server_negotiation.cc:407] Connection header received
0925 05:05:31.040317 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.040318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.040326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.040383 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.041190 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.041661 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.042544 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.042732 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.042788 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.044737 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.044738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.044738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.044745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.046310 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.046797 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.046798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.046798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.046858 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.047177 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.047178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.047179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.047307 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.047376 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.047727 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.047807 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:05:31.048105 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.039682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:34541 (local address 127.14.33.1:34157)
0925 05:05:31.039826 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:31.039860 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:31.040244 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:31.040430 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:31.040433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:31.040895 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:31.041071 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.041075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.041759 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.041760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.042461 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.042463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.042811 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.042811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.042896 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.043338 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:31.043345 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:31.044646 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:31.046399 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.046406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.046408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.046699 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.046931 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.046932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.046933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.047081 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.047390 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:31.047396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:31.047565 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 05:05:31.047675 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:05:31.047847 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260925 05:05:31.048542 14711 heartbeater.cc:347] Connected to a master server at 127.14.33.62:34541
I20260925 05:05:31.048635 14711 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:31.048817 14711 heartbeater.cc:511] Master 127.14.33.62:34541 requested a full tablet report, sending...
I20260925 05:05:31.049360 14520 ts_manager.cc:194] Registered new tserver with Master: 4f1262b1bc3f4d3eaaf849f9c3df3bdf (127.14.33.1:35607)
I20260925 05:05:31.049939 14520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:34157
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:31.068089 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34541
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:31.145017 14716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:31.145303 14716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:31.145350 14716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:31.146502 14716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:31.146564 14716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:31.146610 14716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:31.148160 14716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:31.148414 14716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:31.148655 14716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:31.153599 14724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.153612 14722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.153591 14721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:31.153838 14716 server_base.cc:1034] running on GCE node
I20260925 05:05:31.154029 14716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:31.154274 14716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:31.155445 14716 hybrid_clock.cc:648] HybridClock initialized: now 1790312731155423 us; error 33 us; skew 500 ppm
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:31.162700 14716 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:31.163216 14716 webserver.cc:460] Webserver started at http://127.14.33.2:34039/ using document root <none> and password file <none>
I20260925 05:05:31.163457 14716 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:31.163525 14716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:31.163686 14716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:31.164746 14716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "9458fb28956842f8a6a189b663e914c6"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "439091ef684c654529556771d7792fa2"
server_key_iv: "2b407e3503006fe0aefe8804fe9cb998"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.165095 14716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "9458fb28956842f8a6a189b663e914c6"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "439091ef684c654529556771d7792fa2"
server_key_iv: "2b407e3503006fe0aefe8804fe9cb998"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.166392 14716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:31.167091 14731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.167289 14716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:31.167392 14716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "9458fb28956842f8a6a189b663e914c6"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "439091ef684c654529556771d7792fa2"
server_key_iv: "2b407e3503006fe0aefe8804fe9cb998"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.167472 14716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:31.215777 14716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:31.217613 14716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:31.217799 14716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:31.218039 14716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:31.218353 14716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:31.218407 14716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.218448 14716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:31.218477 14716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.224460 14716 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:33005
I20260925 05:05:31.224500 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:33005 every 8 connection(s)
I20260925 05:05:31.224789 14716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:31.232815 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 14716
I20260925 05:05:31.232913 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:31.233079 14468 external_mini_cluster.cc:1428] Setting key 69babbc542664f6f037f4d5bfd530588
I20260925 05:05:31.233083 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.225418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:50017 (local address 127.14.33.62:34541)
0925 05:05:31.225472 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.225474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.225697 (+ 223us) server_negotiation.cc:407] Connection header received
0925 05:05:31.226148 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.226149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.226159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.226213 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.227086 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.227573 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.228482 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.228665 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.228725 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.230664 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.230665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.230666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.230672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.231817 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.232172 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.232173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.232174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.232230 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.232445 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.232446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.232446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.232545 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.232596 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.232932 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.233013 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:05:31.233289 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.225402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:34541 (local address 127.14.33.2:50017)
0925 05:05:31.225544 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:31.225576 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:31.226067 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:31.226265 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:31.226268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:31.226782 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:31.226960 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.226965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.227663 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.227664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.228407 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.228409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.228734 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.228735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.228846 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.229362 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:31.229370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:31.230586 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:31.231884 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.231889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.231890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.232087 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.232302 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.232303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.232303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.232389 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.232598 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:31.232601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:31.232763 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 05:05:31.232880 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:05:31.233037 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260925 05:05:31.233868 14845 heartbeater.cc:347] Connected to a master server at 127.14.33.62:34541
I20260925 05:05:31.233973 14845 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:31.234138 14845 heartbeater.cc:511] Master 127.14.33.62:34541 requested a full tablet report, sending...
I20260925 05:05:31.234534 14520 ts_manager.cc:194] Registered new tserver with Master: 9458fb28956842f8a6a189b663e914c6 (127.14.33.2:33005)
I20260925 05:05:31.235015 14520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:50017
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:31.255439 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34541
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:31.332479 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:31.332688 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:31.332723 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:31.333797 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:31.333858 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:31.333899 14850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:31.335254 14850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34541
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:31.335515 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:31.335721 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:05:31.340251 14850 server_base.cc:1034] running on GCE node
W20260925 05:05:31.340202 14858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.340198 14855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:31.340252 14856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:31.340646 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:31.340888 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:31.342037 14850 hybrid_clock.cc:648] HybridClock initialized: now 1790312731341997 us; error 56 us; skew 500 ppm
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:31.344312 14850 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:31.344717 14850 webserver.cc:460] Webserver started at http://127.14.33.3:41309/ using document root <none> and password file <none>
I20260925 05:05:31.344930 14850 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:31.344993 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:31.345140 14850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:31.346045 14850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "e7a524f0908b4481be694b9a5002b456"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "9a57305442e78c769271cc8f4df8bd16"
server_key_iv: "14f55c76a0e3a480cd815ffd1c55c6e6"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.346381 14850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a524f0908b4481be694b9a5002b456"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "9a57305442e78c769271cc8f4df8bd16"
server_key_iv: "14f55c76a0e3a480cd815ffd1c55c6e6"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.347731 14850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:31.348402 14865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.348568 14850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:31.348654 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "e7a524f0908b4481be694b9a5002b456"
format_stamp: "Formatted at 2026-09-25 05:05:31 on dist-test-slave-4dmq"
server_key: "9a57305442e78c769271cc8f4df8bd16"
server_key_iv: "14f55c76a0e3a480cd815ffd1c55c6e6"
server_key_version: "encryptionkey@0"
I20260925 05:05:31.348745 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:31.362823 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:31.364454 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:31.364598 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:31.364802 14850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:31.365072 14850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:31.365121 14850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.365231 14850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:31.365262 14850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:31.370786 14850 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:38305
I20260925 05:05:31.370828 14978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:38305 every 8 connection(s)
I20260925 05:05:31.371104 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:31.379266 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.371770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:43589 (local address 127.14.33.62:34541)
0925 05:05:31.371822 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.371823 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.372010 (+ 187us) server_negotiation.cc:407] Connection header received
0925 05:05:31.372410 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.372411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.372421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.372469 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.373382 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.373893 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.374852 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.375039 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.375095 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.376940 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.376941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.376942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.376948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.378122 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.378457 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.378458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.378459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.378508 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.378742 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.378743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.378743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.378864 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.378924 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.379141 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.379203 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:05:31.379436 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 14850
I20260925 05:05:31.379479 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.371762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:34541 (local address 127.14.33.3:43589)
0925 05:05:31.371893 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:31.371923 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:31.372333 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:31.372516 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:31.372519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:31.373035 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:31.373274 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.373278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.373983 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.373984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.374765 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:31.374767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.375111 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.375112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:31.375201 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.375599 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:31.375606 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:31.376856 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:31.378189 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.378194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.378195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.378394 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.378564 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:31.378564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:31.378564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:31.378653 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:31.378926 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:31.378929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:31.379029 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 05:05:31.379108 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:05:31.379215 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260925 05:05:31.379537 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:31.379737 14468 external_mini_cluster.cc:1428] Setting key b07d1a7e68cda65cb85be6a567d2973c
I20260925 05:05:31.379921 14979 heartbeater.cc:347] Connected to a master server at 127.14.33.62:34541
I20260925 05:05:31.380014 14979 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:31.380193 14979 heartbeater.cc:511] Master 127.14.33.62:34541 requested a full tablet report, sending...
I20260925 05:05:31.380587 14519 ts_manager.cc:194] Registered new tserver with Master: e7a524f0908b4481be694b9a5002b456 (127.14.33.3:38305)
I20260925 05:05:31.381006 14519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:43589
I20260925 05:05:31.391546 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:31.412875 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.405976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.14.33.1:35607)
0925 05:05:31.406176 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.406182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.406195 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:05:31.406232 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.406234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.406248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.406313 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.406843 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.407540 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.408216 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.408554 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.408606 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.410680 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.410688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.410688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.410694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.411995 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.412282 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.412284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.412284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.412334 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.412471 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.412471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.412471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.412579 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.412674 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.412725 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.412791 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
W20260925 05:05:31.413246 14625 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37334
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:31.432898 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:31.426858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.14.33.1:35607)
0925 05:05:31.426908 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:31.426910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:31.426997 (+ 87us) server_negotiation.cc:407] Connection header received
0925 05:05:31.427033 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:31.427033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:31.427043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:31.427090 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:31.427568 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.427999 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.428645 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:31.428827 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:31.428871 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:31.430794 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:31.430795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:31.430796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:31.430802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:31.431871 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.432186 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.432188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.432188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.432249 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:31.432437 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:31.432439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:31.432439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:31.432530 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:31.432644 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:31.432728 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 05:05:31.432802 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:05:31.433401 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 14582
I20260925 05:05:31.437994 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 14716
I20260925 05:05:31.443783 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 14850
I20260925 05:05:31.447604 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 14488
2026-09-25T05:05:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1872 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14993](info): set up 2 sockets
Sep 25 05:05:31 dist-test-slave-4dmq krb5kdc[14993](info): commencing operation
krb5kdc: starting...
W20260925 05:05:32.421900 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:32.524233 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:39003
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:32.597883 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:32.598109 15006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:32.598140 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:32.599190 15006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:32.599251 15006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:32.599273 15006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:32.599291 15006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:32.599318 15006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:32.600605 15006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:32.600878 15006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:32.601091 15006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:32.605531 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.605548 15011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.605604 15012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:32.605731 15006 server_base.cc:1034] running on GCE node
I20260925 05:05:32.605965 15006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:32.606238 15006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:32.607395 15006 hybrid_clock.cc:648] HybridClock initialized: now 1790312732607385 us; error 42 us; skew 500 ppm
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:32.609562 15006 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:32.610005 15006 webserver.cc:460] Webserver started at http://127.14.33.62:32827/ using document root <none> and password file <none>
I20260925 05:05:32.610210 15006 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:32.610273 15006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:32.610409 15006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:32.611342 15006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "91f420e65789409e8ccfd2b0e897d540"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "d8512f51a17fef269f0b3ea24196b2b1"
server_key_iv: "9a4b025ad324101581349ab711a81978"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.611677 15006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "91f420e65789409e8ccfd2b0e897d540"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "d8512f51a17fef269f0b3ea24196b2b1"
server_key_iv: "9a4b025ad324101581349ab711a81978"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.612912 15006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:05:32.613664 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.613843 15006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.613934 15006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "91f420e65789409e8ccfd2b0e897d540"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "d8512f51a17fef269f0b3ea24196b2b1"
server_key_iv: "9a4b025ad324101581349ab711a81978"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.614015 15006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:32.623889 15006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:32.625687 15006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:32.625840 15006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:32.629526 15006 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:39003
I20260925 05:05:32.629555 15073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:39003 every 8 connection(s)
I20260925 05:05:32.629878 15006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:32.630631 15074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:32.632596 15074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540: Bootstrap starting.
I20260925 05:05:32.633078 15074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:32.633366 15074 log.cc:824] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:32.633948 15074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540: No bootstrap required, opened a new log
I20260925 05:05:32.635329 15074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f420e65789409e8ccfd2b0e897d540" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39003 } }
I20260925 05:05:32.635459 15074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:32.635479 15074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f420e65789409e8ccfd2b0e897d540, State: Initialized, Role: FOLLOWER
I20260925 05:05:32.635597 15074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f420e65789409e8ccfd2b0e897d540" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39003 } }
I20260925 05:05:32.635669 15074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:32.635699 15074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:32.635736 15074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:32.636217 15074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f420e65789409e8ccfd2b0e897d540" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39003 } }
I20260925 05:05:32.636296 15074 leader_election.cc:302] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f420e65789409e8ccfd2b0e897d540; no voters:
I20260925 05:05:32.636406 15074 leader_election.cc:288] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:32.636468 15077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:32.636641 15077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [term 1 LEADER]: Becoming Leader. State: Replica: 91f420e65789409e8ccfd2b0e897d540, State: Running, Role: LEADER
I20260925 05:05:32.636662 15074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:32.636734 15077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f420e65789409e8ccfd2b0e897d540" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39003 } }
I20260925 05:05:32.637285 15078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f420e65789409e8ccfd2b0e897d540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f420e65789409e8ccfd2b0e897d540" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39003 } } }
I20260925 05:05:32.637439 15079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f420e65789409e8ccfd2b0e897d540. Latest consensus state: current_term: 1 leader_uuid: "91f420e65789409e8ccfd2b0e897d540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f420e65789409e8ccfd2b0e897d540" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39003 } } }
I20260925 05:05:32.637521 15078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:32.637751 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15006
I20260925 05:05:32.637840 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:32.638006 14468 external_mini_cluster.cc:1428] Setting key f27b057b8b55c50cb52114886bbc989b
I20260925 05:05:32.637965 15089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:32.637581 15079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:32.638785 15089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:32.640885 15089 catalog_manager.cc:1347] Generated new cluster ID: 4c97cbb1db084ca8ac593a938601903a
I20260925 05:05:32.640935 15089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:32.648394 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.638466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.14.33.62:39003)
0925 05:05:32.638613 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.638622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.638653 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:05:32.638774 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.638780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.638844 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.638977 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:32.639666 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.640754 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.641741 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.642350 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.642460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.644950 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:32.644953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:32.644954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:32.644964 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:32.646824 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.647237 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.647239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.647239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.647343 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.647546 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.647547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.647547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.647810 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:32.647938 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.648085 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.648203 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260925 05:05:32.650540 15089 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:32.650983 15089 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:32.657992 15089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f420e65789409e8ccfd2b0e897d540: Generated new TSK 0
I20260925 05:05:32.658149 15089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:32.682109 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39003
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:32.756680 15100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:32.756867 15100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:32.756889 15100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:32.758050 15100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:32.758090 15100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:32.758141 15100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:32.759512 15100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:32.759743 15100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:32.759991 15100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:32.764638 15106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.764680 15105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.764652 15108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:32.764809 15100 server_base.cc:1034] running on GCE node
I20260925 05:05:32.765098 15100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:32.765342 15100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:32.766474 15100 hybrid_clock.cc:648] HybridClock initialized: now 1790312732766448 us; error 45 us; skew 500 ppm
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:32.768814 15100 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:32.769344 15100 webserver.cc:460] Webserver started at http://127.14.33.1:33671/ using document root <none> and password file <none>
I20260925 05:05:32.769559 15100 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:32.769625 15100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:32.769783 15100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:32.770735 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "5d308aec582a4fd8b1ca411b424f4e46"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "b284176d4c1eb73aab163e23a0dbfb02"
server_key_iv: "3b2d49bfc1b8e99e702920c13fdae995"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.771090 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "5d308aec582a4fd8b1ca411b424f4e46"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "b284176d4c1eb73aab163e23a0dbfb02"
server_key_iv: "3b2d49bfc1b8e99e702920c13fdae995"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.772363 15100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:32.773044 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.773218 15100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:32.773311 15100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "5d308aec582a4fd8b1ca411b424f4e46"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "b284176d4c1eb73aab163e23a0dbfb02"
server_key_iv: "3b2d49bfc1b8e99e702920c13fdae995"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.773394 15100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:32.798193 15100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:32.799685 15100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:32.799851 15100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:32.800046 15100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:32.800323 15100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:32.800369 15100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.800403 15100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:32.800427 15100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.805980 15100 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:46593
I20260925 05:05:32.806042 15228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:46593 every 8 connection(s)
I20260925 05:05:32.806360 15100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:32.815078 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.806975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:35973 (local address 127.14.33.62:39003)
0925 05:05:32.807016 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.807018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.807229 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:05:32.808163 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.808164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.808180 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.808230 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:32.808839 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.809393 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.810383 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.810593 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.810666 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.812472 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:32.812473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:32.812474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:32.812480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:32.813721 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.814122 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.814124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.814124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.814182 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.814453 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.814454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.814455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.814565 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:32.814641 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.814906 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.814991 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:05:32.815234 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.806957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39003 (local address 127.14.33.1:35973)
0925 05:05:32.807093 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.807124 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.808056 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.808326 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.808329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.808562 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:32.808733 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.808736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.809478 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.809479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.810256 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.810259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.810684 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.810686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.810800 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.811199 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:32.811219 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:32.812386 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:32.813820 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.813827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.813829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.814033 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.814280 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.814281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.814282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.814383 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.814633 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:32.814640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:32.814773 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.814871 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.815015 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260925 05:05:32.815713 15229 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39003
I20260925 05:05:32.815783 15229 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:32.815940 15229 heartbeater.cc:511] Master 127.14.33.62:39003 requested a full tablet report, sending...
I20260925 05:05:32.816182 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15100
I20260925 05:05:32.816277 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:32.816430 14468 external_mini_cluster.cc:1428] Setting key 98ae3d4766349d10813c14098af1d128
I20260925 05:05:32.816530 15038 ts_manager.cc:194] Registered new tserver with Master: 5d308aec582a4fd8b1ca411b424f4e46 (127.14.33.1:46593)
I20260925 05:05:32.817401 15038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:35973
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:32.837963 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39003
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:32.912889 15234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:32.913064 15234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:32.913085 15234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:32.914189 15234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:32.914230 15234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:32.914256 15234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:32.915545 15234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:32.915692 15234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:32.915887 15234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:32.921103 15239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:32.921140 15240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:32.921147 15234 server_base.cc:1034] running on GCE node
W20260925 05:05:32.921298 15242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:32.921571 15234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:32.921775 15234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:32.922937 15234 hybrid_clock.cc:648] HybridClock initialized: now 1790312732922927 us; error 26 us; skew 500 ppm
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:32.925269 15234 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:32.925706 15234 webserver.cc:460] Webserver started at http://127.14.33.2:45379/ using document root <none> and password file <none>
I20260925 05:05:32.925908 15234 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:32.925968 15234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:32.926093 15234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:32.926962 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "6b62e8dd236642eb9a18093b04ae8cdf"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "02f03dd76ccfc43502639699d7e6cb28"
server_key_iv: "c92d498c2313fd773612d9d47d37730f"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.927309 15234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "6b62e8dd236642eb9a18093b04ae8cdf"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "02f03dd76ccfc43502639699d7e6cb28"
server_key_iv: "c92d498c2313fd773612d9d47d37730f"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.928460 15234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:32.929059 15249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.929211 15234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:32.929315 15234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "6b62e8dd236642eb9a18093b04ae8cdf"
format_stamp: "Formatted at 2026-09-25 05:05:32 on dist-test-slave-4dmq"
server_key: "02f03dd76ccfc43502639699d7e6cb28"
server_key_iv: "c92d498c2313fd773612d9d47d37730f"
server_key_version: "encryptionkey@0"
I20260925 05:05:32.929392 15234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:32.935343 15234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:32.937458 15234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:32.937605 15234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:32.937822 15234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:32.938153 15234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:32.938202 15234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.938237 15234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:32.938261 15234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:32.943652 15234 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:36123
I20260925 05:05:32.943684 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:36123 every 8 connection(s)
I20260925 05:05:32.943998 15234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:05:32 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:32.951915 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15234
I20260925 05:05:32.951993 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.944610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:48243 (local address 127.14.33.62:39003)
0925 05:05:32.944678 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:32.944680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:32.944825 (+ 145us) server_negotiation.cc:407] Connection header received
0925 05:05:32.945629 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:32.945630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:32.945639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:32.945685 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:32.946143 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.946583 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.947408 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.947570 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.947615 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.949349 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:32.949350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:32.949350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:32.949357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:32.950728 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.951068 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.951069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.951070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.951134 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:32.951381 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:32.951382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:32.951382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:32.951495 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:32.951560 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:32.951824 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 05:05:32.951906 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:05:32.952015 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:32.952122 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:32.944577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39003 (local address 127.14.33.2:48243)
0925 05:05:32.944724 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:32.944752 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:32.945550 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:32.945728 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:32.945731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:32.945931 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:32.946081 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.946084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.946650 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.946650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.947343 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:32.947344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:32.947632 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:32.947633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:32.947714 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:32.948039 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:32.948045 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:32.949264 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:32.950795 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.950800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.950801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.950979 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.951199 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:32.951200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:32.951200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:32.951302 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:32.951562 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:32.951568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:32.951681 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 05:05:32.951784 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:05:32.951921 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260925 05:05:32.952175 14468 external_mini_cluster.cc:1428] Setting key 28da17fd46e5ee1f2849bcb3fdcce102
I20260925 05:05:32.952591 15363 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39003
I20260925 05:05:32.952677 15363 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:32.952829 15363 heartbeater.cc:511] Master 127.14.33.62:39003 requested a full tablet report, sending...
I20260925 05:05:32.953223 15037 ts_manager.cc:194] Registered new tserver with Master: 6b62e8dd236642eb9a18093b04ae8cdf (127.14.33.2:36123)
I20260925 05:05:32.953593 15037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:48243
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:32.973271 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39003
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:33.047629 15368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:33.047817 15368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:33.047838 15368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:33.048906 15368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:33.048938 15368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:33.048964 15368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:33.050297 15368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:33.050475 15368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:33.050622 15368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:33.055326 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:33.055361 15376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:33.055482 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:33.055464 15368 server_base.cc:1034] running on GCE node
I20260925 05:05:33.055730 15368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:33.055933 15368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:33.057067 15368 hybrid_clock.cc:648] HybridClock initialized: now 1790312733057061 us; error 33 us; skew 500 ppm
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312733, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:33.059290 15368 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:33.059677 15368 webserver.cc:460] Webserver started at http://127.14.33.3:36353/ using document root <none> and password file <none>
I20260925 05:05:33.059890 15368 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:33.059932 15368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:33.060062 15368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:33.061022 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d"
format_stamp: "Formatted at 2026-09-25 05:05:33 on dist-test-slave-4dmq"
server_key: "38d3fd7af30f63e1a63bff4e7795568e"
server_key_iv: "1d515608816de5db961bd84c299ea478"
server_key_version: "encryptionkey@0"
I20260925 05:05:33.061415 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d"
format_stamp: "Formatted at 2026-09-25 05:05:33 on dist-test-slave-4dmq"
server_key: "38d3fd7af30f63e1a63bff4e7795568e"
server_key_iv: "1d515608816de5db961bd84c299ea478"
server_key_version: "encryptionkey@0"
I20260925 05:05:33.062595 15368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:33.063272 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:33.063457 15368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:33.063535 15368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d"
format_stamp: "Formatted at 2026-09-25 05:05:33 on dist-test-slave-4dmq"
server_key: "38d3fd7af30f63e1a63bff4e7795568e"
server_key_iv: "1d515608816de5db961bd84c299ea478"
server_key_version: "encryptionkey@0"
I20260925 05:05:33.063603 15368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:33.077493 15368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:33.079092 15368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:33.079242 15368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:33.079440 15368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:33.079718 15368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:33.079751 15368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:33.079770 15368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:33.079823 15368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:33.085242 15368 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:46813
I20260925 05:05:33.085312 15496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:46813 every 8 connection(s)
I20260925 05:05:33.085594 15368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:33.087175 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15368
I20260925 05:05:33.087327 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:33.087512 14468 external_mini_cluster.cc:1428] Setting key 12f9d750d92549cb8c11d5645dbf7ca4
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312733, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:33.094496 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.086273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:56435 (local address 127.14.33.62:39003)
0925 05:05:33.086338 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.086340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.086511 (+ 171us) server_negotiation.cc:407] Connection header received
0925 05:05:33.087705 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.087706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.087720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.087772 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.088512 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.088991 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.089995 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.090193 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.090283 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.092239 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.092241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.092241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.092248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.093447 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.093781 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.093782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.093782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.093830 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.094033 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.094033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.094033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.094123 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.094169 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.094380 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.094436 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:05:33.094659 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.086247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39003 (local address 127.14.33.3:56435)
0925 05:05:33.086393 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:33.086434 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:33.087583 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:33.087872 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:33.087875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:33.088172 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:33.088396 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.088407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.089085 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.089086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.089891 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.089893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.090291 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.090292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.090421 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.090904 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:33.090911 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:33.092155 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:33.093513 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:33.093518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:33.093520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:33.093720 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:33.093884 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:33.093884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:33.093884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:33.093979 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:33.094173 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:33.094177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:33.094279 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 05:05:33.094352 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:05:33.094455 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260925 05:05:33.095191 15497 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39003
I20260925 05:05:33.095306 15497 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:33.095469 15497 heartbeater.cc:511] Master 127.14.33.62:39003 requested a full tablet report, sending...
I20260925 05:05:33.095798 15037 ts_manager.cc:194] Registered new tserver with Master: 8fd99a2881fc4e80aaf0e8e24230dd7d (127.14.33.3:46813)
I20260925 05:05:33.096128 15037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:56435
I20260925 05:05:33.098992 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:33.105592 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.100432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.14.33.62:39003)
0925 05:05:33.100486 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.100487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.100589 (+ 102us) server_negotiation.cc:407] Connection header received
0925 05:05:33.100627 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.100628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.100635 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.100677 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.101262 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.101670 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.102327 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.102490 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.102531 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.103432 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.103433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.103433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.103438 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.104707 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.104963 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.104964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.104964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.105004 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.105112 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.105113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.105113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.105273 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.105354 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.105449 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.105499 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:05:33.107695 15037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:33.109210 15037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:33.115041 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.111068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:46813 (local address 127.0.0.1:34940)
0925 05:05:33.111240 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:33.111279 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:33.111340 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:33.111702 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:33.111705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:33.111707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:33.111775 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.111779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.112681 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.112682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.113919 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.113922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.114761 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.114836 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.114848 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:05:33.114924 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 05:05:33.114978 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":5,"thread_start_us":112,"threads_started":1}
I20260925 05:05:33.115126 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.111179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.14.33.1:46593)
0925 05:05:33.111376 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.111381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.111515 (+ 134us) server_negotiation.cc:407] Connection header received
0925 05:05:33.111570 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.111572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.111584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.111693 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:33.111853 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.112927 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114202 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.114678 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114766 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.114821 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.115013 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.115055 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260925 05:05:33.115386 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.111090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:36123 (local address 127.0.0.1:36490)
0925 05:05:33.111576 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:33.111584 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:33.111630 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:33.111873 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:33.111873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:33.111875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:33.111908 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.111909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.113258 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.113259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.114205 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.114208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.115162 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.115163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.115217 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.115221 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:05:33.115274 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:05:33.115312 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":87,"threads_started":1}
I20260925 05:05:33.115425 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.111379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.14.33.3:46813)
0925 05:05:33.111534 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.111538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.111547 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:05:33.111581 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.111584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.111600 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.111854 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:33.111912 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.112567 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114004 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.114676 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114750 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.114799 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.114950 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.115351 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260925 05:05:33.115520 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.111069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:46593 (local address 127.0.0.1:56648)
0925 05:05:33.111444 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:33.111452 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:33.111506 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:33.111710 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:33.111710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:33.111712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:33.111775 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.111777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.113058 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.113060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.114105 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.114107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.114875 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.114931 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.114933 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:33.115335 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 05:05:33.115378 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":3,"thread_start_us":45,"threads_started":1}
I20260925 05:05:33.115643 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.111309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.14.33.2:36123)
0925 05:05:33.111468 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.111472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.111646 (+ 174us) server_negotiation.cc:407] Connection header received
0925 05:05:33.111694 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.111696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.111713 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.111800 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:33.112061 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.113106 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.114316 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.114998 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.115065 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.115119 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.115517 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.115563 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260925 05:05:33.116083 15431 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed382ee084c4bb9b8c019009ac4f647 (DEFAULT_TABLE table=test-table [id=839b68a7ee704ea48b6f46ba9827e1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:33.116194 15297 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed382ee084c4bb9b8c019009ac4f647 (DEFAULT_TABLE table=test-table [id=839b68a7ee704ea48b6f46ba9827e1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:33.116418 15431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed382ee084c4bb9b8c019009ac4f647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:33.116573 15297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed382ee084c4bb9b8c019009ac4f647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:33.118940 15511 tablet_bootstrap.cc:492] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: Bootstrap starting.
I20260925 05:05:33.119423 15512 tablet_bootstrap.cc:492] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: Bootstrap starting.
I20260925 05:05:33.119356 15163 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed382ee084c4bb9b8c019009ac4f647 (DEFAULT_TABLE table=test-table [id=839b68a7ee704ea48b6f46ba9827e1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:33.119604 15511 tablet_bootstrap.cc:654] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:33.119634 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed382ee084c4bb9b8c019009ac4f647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:33.119982 15512 tablet_bootstrap.cc:654] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:33.120103 15511 log.cc:824] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:33.120604 15512 log.cc:824] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:33.121505 15512 tablet_bootstrap.cc:492] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: No bootstrap required, opened a new log
I20260925 05:05:33.121600 15512 ts_tablet_manager.cc:1397] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:33.121759 15511 tablet_bootstrap.cc:492] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: No bootstrap required, opened a new log
I20260925 05:05:33.121835 15511 ts_tablet_manager.cc:1397] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:05:33.122874 15515 tablet_bootstrap.cc:492] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: Bootstrap starting.
I20260925 05:05:33.123533 15515 tablet_bootstrap.cc:654] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:33.123414 15512 raft_consensus.cc:348] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.123588 15512 raft_consensus.cc:374] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:33.123631 15512 raft_consensus.cc:729] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b62e8dd236642eb9a18093b04ae8cdf, State: Initialized, Role: FOLLOWER
I20260925 05:05:33.123571 15511 raft_consensus.cc:348] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.123793 15511 raft_consensus.cc:374] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:33.123839 15511 raft_consensus.cc:729] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd99a2881fc4e80aaf0e8e24230dd7d, State: Initialized, Role: FOLLOWER
I20260925 05:05:33.123832 15512 consensus_queue.cc:260] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.123965 15511 consensus_queue.cc:260] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.124137 15515 log.cc:824] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:33.124177 15363 heartbeater.cc:503] Master 127.14.33.62:39003 was elected leader, sending a full tablet report...
I20260925 05:05:33.124226 15511 ts_tablet_manager.cc:1428] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:05:33.124332 15497 heartbeater.cc:503] Master 127.14.33.62:39003 was elected leader, sending a full tablet report...
I20260925 05:05:33.124445 15512 ts_tablet_manager.cc:1428] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:05:33.125166 15515 tablet_bootstrap.cc:492] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: No bootstrap required, opened a new log
I20260925 05:05:33.125244 15515 ts_tablet_manager.cc:1397] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:33.126523 15515 raft_consensus.cc:348] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.126688 15515 raft_consensus.cc:374] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:33.126729 15515 raft_consensus.cc:729] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d308aec582a4fd8b1ca411b424f4e46, State: Initialized, Role: FOLLOWER
I20260925 05:05:33.126827 15515 consensus_queue.cc:260] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.127019 15515 ts_tablet_manager.cc:1428] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:33.127063 15229 heartbeater.cc:503] Master 127.14.33.62:39003 was elected leader, sending a full tablet report...
W20260925 05:05:33.194655 15364 tablet.cc:2369] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:33.307037 15230 tablet.cc:2369] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:33.336114 15498 tablet.cc:2369] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:33.342572 15519 raft_consensus.cc:482] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:33.342691 15519 raft_consensus.cc:504] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.343045 15519 leader_election.cc:288] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd99a2881fc4e80aaf0e8e24230dd7d (127.14.33.3:46813), 6b62e8dd236642eb9a18093b04ae8cdf (127.14.33.2:36123)
I20260925 05:05:33.345980 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.343312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:38327 (local address 127.14.33.3:46813)
0925 05:05:33.343358 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.343360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.343464 (+ 104us) server_negotiation.cc:407] Connection header received
0925 05:05:33.343537 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.343538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.343547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.343613 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:33.343783 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.344235 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.345247 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.345623 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.345704 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.345721 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.345873 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.345909 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:05:33.345988 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.343312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:50373 (local address 127.14.33.2:36123)
0925 05:05:33.343361 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.343363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.343405 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:05:33.343497 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.343498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.343505 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.343576 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:33.343725 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.344193 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.345247 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.345623 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.345705 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.345715 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.345887 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.345921 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:05:33.346004 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.343228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:46813 (local address 127.14.33.1:38327)
0925 05:05:33.343381 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:33.343391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:33.343476 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:33.343616 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:33.343617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:33.343619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:33.343710 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.343712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.344315 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.344316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.345102 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.345104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.345704 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.345705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.345781 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.345784 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:33.345868 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:05:33.345912 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260925 05:05:33.346014 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.343228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:36123 (local address 127.14.33.1:50373)
0925 05:05:33.343348 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:33.343359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:33.343420 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:33.343579 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:33.343580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:33.343582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:33.343654 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.343656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.344275 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.344276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.345099 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:33.345101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.345707 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.345708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:33.345787 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.345791 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:05:33.345874 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:05:33.345916 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":44,"thread_start_us":65,"threads_started":1}
I20260925 05:05:33.346405 15451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed382ee084c4bb9b8c019009ac4f647" candidate_uuid: "5d308aec582a4fd8b1ca411b424f4e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" is_pre_election: true
I20260925 05:05:33.346558 15451 raft_consensus.cc:2454] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d308aec582a4fd8b1ca411b424f4e46 in term 0.
I20260925 05:05:33.346657 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed382ee084c4bb9b8c019009ac4f647" candidate_uuid: "5d308aec582a4fd8b1ca411b424f4e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" is_pre_election: true
I20260925 05:05:33.346799 15317 raft_consensus.cc:2454] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d308aec582a4fd8b1ca411b424f4e46 in term 0.
I20260925 05:05:33.346798 15116 leader_election.cc:302] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d308aec582a4fd8b1ca411b424f4e46, 8fd99a2881fc4e80aaf0e8e24230dd7d; no voters:
I20260925 05:05:33.346967 15519 raft_consensus.cc:2781] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:33.347025 15519 raft_consensus.cc:482] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:33.347048 15519 raft_consensus.cc:3037] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:33.347680 15519 raft_consensus.cc:504] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.347815 15519 leader_election.cc:288] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [CANDIDATE]: Term 1 election: Requested vote from peers 8fd99a2881fc4e80aaf0e8e24230dd7d (127.14.33.3:46813), 6b62e8dd236642eb9a18093b04ae8cdf (127.14.33.2:36123)
I20260925 05:05:33.347965 15451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed382ee084c4bb9b8c019009ac4f647" candidate_uuid: "5d308aec582a4fd8b1ca411b424f4e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d"
I20260925 05:05:33.347975 15317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed382ee084c4bb9b8c019009ac4f647" candidate_uuid: "5d308aec582a4fd8b1ca411b424f4e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b62e8dd236642eb9a18093b04ae8cdf"
I20260925 05:05:33.348042 15451 raft_consensus.cc:3037] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:33.348042 15317 raft_consensus.cc:3037] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:33.348875 15317 raft_consensus.cc:2454] T 1ed382ee084c4bb9b8c019009ac4f647 P 6b62e8dd236642eb9a18093b04ae8cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d308aec582a4fd8b1ca411b424f4e46 in term 1.
I20260925 05:05:33.348875 15451 raft_consensus.cc:2454] T 1ed382ee084c4bb9b8c019009ac4f647 P 8fd99a2881fc4e80aaf0e8e24230dd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d308aec582a4fd8b1ca411b424f4e46 in term 1.
I20260925 05:05:33.349045 15118 leader_election.cc:302] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d308aec582a4fd8b1ca411b424f4e46, 6b62e8dd236642eb9a18093b04ae8cdf; no voters:
I20260925 05:05:33.349172 15519 raft_consensus.cc:2781] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:33.349318 15519 raft_consensus.cc:686] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [term 1 LEADER]: Becoming Leader. State: Replica: 5d308aec582a4fd8b1ca411b424f4e46, State: Running, Role: LEADER
I20260925 05:05:33.349419 15519 consensus_queue.cc:237] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } }
I20260925 05:05:33.350178 15037 catalog_manager.cc:5495] T 1ed382ee084c4bb9b8c019009ac4f647 P 5d308aec582a4fd8b1ca411b424f4e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d308aec582a4fd8b1ca411b424f4e46 (127.14.33.1). New cstate: current_term: 1 leader_uuid: "5d308aec582a4fd8b1ca411b424f4e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd99a2881fc4e80aaf0e8e24230dd7d" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d308aec582a4fd8b1ca411b424f4e46" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b62e8dd236642eb9a18093b04ae8cdf" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 36123 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:33.401746 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.394383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.14.33.1:46593)
0925 05:05:33.394488 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.394490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.394563 (+ 73us) server_negotiation.cc:407] Connection header received
0925 05:05:33.394644 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.394645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.394659 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.394714 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.395521 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.396033 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.396707 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.396906 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.396976 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.399068 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.399080 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.399080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.399087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.400807 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.401069 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.401071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.401072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.401225 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.401369 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.401369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.401369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.401490 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.401591 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.401624 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.401686 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:33.422916 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.416235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.14.33.1:46593)
0925 05:05:33.416292 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.416294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.416376 (+ 82us) server_negotiation.cc:407] Connection header received
0925 05:05:33.416463 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.416464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.416481 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.416547 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.417073 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.417583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.418296 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.418494 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.418579 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.420580 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.420581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.420581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.420588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.421744 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.422097 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.422098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.422098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.422168 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.422373 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.422374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.422374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.422468 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.422586 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.422749 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.422821 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[14993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:33.442974 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:33.436588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.14.33.1:46593)
0925 05:05:33.436651 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:33.436653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:33.436703 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:05:33.436766 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:33.436767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:33.436774 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:33.436817 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:33.437442 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.437866 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.438688 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:33.438863 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:33.438914 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:33.440865 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:33.440867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:33.440867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:33.440872 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:33.442131 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.442415 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.442416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.442416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.442458 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:33.442612 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:33.442612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:33.442613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:33.442686 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:33.442770 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:33.442848 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:05:33.442896 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 05:05:33.443430 15143 tablet_service.cc:2908] Rejecting scan request for tablet 1ed382ee084c4bb9b8c019009ac4f647: Uninitialized: clean time has not yet been initialized
I20260925 05:05:33.444260 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15100
I20260925 05:05:33.450595 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15234
I20260925 05:05:33.454835 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15368
I20260925 05:05:33.458976 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15006
2026-09-25T05:05:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2011 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[15536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[15536](info): set up 2 sockets
Sep 25 05:05:33 dist-test-slave-4dmq krb5kdc[15536](info): commencing operation
krb5kdc: starting...
W20260925 05:05:34.437939 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:34.543695 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40783
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:34.618417 15549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:34.618595 15549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:34.618616 15549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:34.619726 15549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:34.619768 15549 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:05:34.619781 15549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:34.619794 15549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:34.619828 15549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:34.619856 15549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:34.621368 15549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:34.621598 15549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:34.621811 15549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:34.626427 15557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.626410 15555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.626400 15554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.626641 15549 server_base.cc:1034] running on GCE node
I20260925 05:05:34.626835 15549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:34.627113 15549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:34.628278 15549 hybrid_clock.cc:648] HybridClock initialized: now 1790312734628273 us; error 38 us; skew 500 ppm
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:34.630518 15549 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:34.630951 15549 webserver.cc:460] Webserver started at http://127.14.33.62:45429/ using document root <none> and password file <none>
I20260925 05:05:34.631127 15549 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:34.631165 15549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:34.631247 15549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:34.632135 15549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "b93fb05b8c214570b4a738127e004d0e"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "27cecd0824574b25c040a9a93c28fe23"
server_key_iv: "f710285fb32994deb268448493b82775"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.632418 15549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "b93fb05b8c214570b4a738127e004d0e"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "27cecd0824574b25c040a9a93c28fe23"
server_key_iv: "f710285fb32994deb268448493b82775"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.633680 15549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:34.634344 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.634516 15549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:34.634591 15549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "b93fb05b8c214570b4a738127e004d0e"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "27cecd0824574b25c040a9a93c28fe23"
server_key_iv: "f710285fb32994deb268448493b82775"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.634640 15549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:34.646621 15549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:34.648154 15549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:34.648248 15549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:34.652347 15549 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40783
I20260925 05:05:34.652422 15626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40783 every 8 connection(s)
I20260925 05:05:34.652725 15549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:34.653412 15627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:34.655515 15627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: Bootstrap starting.
I20260925 05:05:34.656073 15627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:34.656355 15627 log.cc:824] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:34.656986 15627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: No bootstrap required, opened a new log
I20260925 05:05:34.657526 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15549
I20260925 05:05:34.657618 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:34.657775 14468 external_mini_cluster.cc:1428] Setting key 0de4e7220e7d610fea6a83831602d409
I20260925 05:05:34.659722 15627 raft_consensus.cc:348] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93fb05b8c214570b4a738127e004d0e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40783 } }
I20260925 05:05:34.659879 15627 raft_consensus.cc:374] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:34.659916 15627 raft_consensus.cc:729] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93fb05b8c214570b4a738127e004d0e, State: Initialized, Role: FOLLOWER
I20260925 05:05:34.660033 15627 consensus_queue.cc:260] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93fb05b8c214570b4a738127e004d0e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40783 } }
I20260925 05:05:34.660107 15627 raft_consensus.cc:388] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:34.660128 15627 raft_consensus.cc:482] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:34.660189 15627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:34.660768 15627 raft_consensus.cc:504] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93fb05b8c214570b4a738127e004d0e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40783 } }
I20260925 05:05:34.660866 15627 leader_election.cc:302] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b93fb05b8c214570b4a738127e004d0e; no voters:
I20260925 05:05:34.661012 15627 leader_election.cc:288] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:34.661067 15632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:34.661229 15632 raft_consensus.cc:686] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [term 1 LEADER]: Becoming Leader. State: Replica: b93fb05b8c214570b4a738127e004d0e, State: Running, Role: LEADER
I20260925 05:05:34.661255 15627 sys_catalog.cc:564] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:34.662021 15632 consensus_queue.cc:237] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93fb05b8c214570b4a738127e004d0e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40783 } }
I20260925 05:05:34.662694 15633 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b93fb05b8c214570b4a738127e004d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93fb05b8c214570b4a738127e004d0e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40783 } } }
I20260925 05:05:34.663084 15633 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:34.662842 15634 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b93fb05b8c214570b4a738127e004d0e. Latest consensus state: current_term: 1 leader_uuid: "b93fb05b8c214570b4a738127e004d0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93fb05b8c214570b4a738127e004d0e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40783 } } }
I20260925 05:05:34.663230 15634 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e [sys.catalog]: This master's current role is: LEADER
W20260925 05:05:34.663477 15646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:05:34.663522 15646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:05:34.663576 15649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:34.663710 15649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:34.665378 15649 catalog_manager.cc:1347] Generated new cluster ID: dc32792be224464fbdab65c44e54da6d
I20260925 05:05:34.665452 15649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:34.670861 15649 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:34.671530 15649 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:34.673822 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.658254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.14.33.62:40783)
0925 05:05:34.658432 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.658441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.658471 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:05:34.659683 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.659690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.659750 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.659892 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.660582 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.661872 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.662898 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.663458 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.663607 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.670246 (+ 6639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.670250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.670250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:34.670261 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.672311 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.672693 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.672696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.672696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.672797 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.672996 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.672996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.672997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.673246 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.673380 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.673488 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.673610 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260925 05:05:34.677332 15649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: Generated new TSK 0
I20260925 05:05:34.677492 15649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:34.701610 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40783
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:34.795331 15654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:34.795509 15654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:34.795529 15654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:34.796627 15654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:34.796664 15654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:34.796684 15654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:05:34.796700 15654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:34.798086 15654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:40783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:34.798308 15654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:34.798547 15654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:34.804369 15660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.804335 15662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.804421 15659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.804627 15654 server_base.cc:1034] running on GCE node
I20260925 05:05:34.804803 15654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:34.805047 15654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:34.806209 15654 hybrid_clock.cc:648] HybridClock initialized: now 1790312734806201 us; error 35 us; skew 500 ppm
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:34.808744 15654 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:34.809197 15654 webserver.cc:460] Webserver started at http://127.14.33.1:44899/ using document root <none> and password file <none>
I20260925 05:05:34.809417 15654 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:34.809460 15654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:34.809608 15654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:34.810583 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "184fbd28dfa7400e90eb4596785a52b5"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "54913e693959d8ddfaee01c3ae82e42b"
server_key_iv: "731a5ecc54fb1cdae5d75364643e4e64"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.810927 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "184fbd28dfa7400e90eb4596785a52b5"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "54913e693959d8ddfaee01c3ae82e42b"
server_key_iv: "731a5ecc54fb1cdae5d75364643e4e64"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.812201 15654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:34.812918 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.813087 15654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:34.813194 15654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "184fbd28dfa7400e90eb4596785a52b5"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "54913e693959d8ddfaee01c3ae82e42b"
server_key_iv: "731a5ecc54fb1cdae5d75364643e4e64"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.813271 15654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:34.820904 15654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:34.822633 15654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:34.822788 15654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:34.823242 15654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:34.823320 15654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.823360 15654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:34.823386 15654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.829813 15654 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:45319
I20260925 05:05:34.829838 15784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:45319 every 8 connection(s)
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:34.830164 15654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:34.832783 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.823734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:33581 (local address 127.14.33.62:40783)
0925 05:05:34.823812 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.823814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.823982 (+ 168us) server_negotiation.cc:407] Connection header received
0925 05:05:34.824373 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.824375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.824391 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.824462 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.825211 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.826019 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.827534 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.827888 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.827967 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.830497 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.830499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.830499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:34.830507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.831732 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.832076 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.832077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.832077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.832125 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.832335 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.832336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.832336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.832441 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.832486 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.832664 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.832728 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:05:34.832914 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.823662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.14.33.1:33581)
0925 05:05:34.823822 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:34.823856 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:34.824288 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:34.824510 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:34.824513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:34.824847 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:34.825111 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.825116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.826122 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.826123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.827433 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.827436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.827982 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.827983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.828117 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.828632 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:34.828641 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:34.830400 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:34.831800 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.831805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.831807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.832009 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.832181 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.832181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.832181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.832285 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.832491 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:34.832495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:34.832587 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 05:05:34.832656 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:05:34.832750 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260925 05:05:34.833391 15785 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40783
I20260925 05:05:34.833472 15785 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:34.833626 15785 heartbeater.cc:511] Master 127.14.33.62:40783 requested a full tablet report, sending...
I20260925 05:05:34.834069 15580 ts_manager.cc:194] Registered new tserver with Master: 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319)
I20260925 05:05:34.834860 15580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:33581
I20260925 05:05:34.835872 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15654
I20260925 05:05:34.835964 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:34.836138 14468 external_mini_cluster.cc:1428] Setting key 7ebb14431373f2f7d0c42be984a8ce01
I20260925 05:05:34.839700 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.834310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.14.33.1:55133)
0925 05:05:34.834451 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:34.834467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:34.834523 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:34.834623 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:34.834623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:34.834779 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:34.834822 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.834823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.835474 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.835475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.836110 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.836113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.836552 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.836553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.836622 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.837050 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:34.837054 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:34.837367 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:34.838925 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.838925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.838926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.839072 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.839261 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.839261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.839261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.839306 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.839523 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:34.839524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:34.839552 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 05:05:34.839589 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 05:05:34.839649 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260925 05:05:34.839792 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.834374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:55133 (local address 127.14.33.62:40783)
0925 05:05:34.834412 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.834414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.834531 (+ 117us) server_negotiation.cc:407] Connection header received
0925 05:05:34.834553 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.834554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.834563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.834620 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.834883 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.835385 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.836201 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.836461 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.836553 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.837448 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.837450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.837450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:34.837456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.838846 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.839138 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.839140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.839140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.839193 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.839353 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.839354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.839354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.839471 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.839532 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.839618 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.839684 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:34.857741 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40783
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:34.929921 15794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:34.930084 15794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:34.930104 15794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:34.931269 15794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:34.931308 15794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:34.931326 15794 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:05:34.931383 15794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:34.932718 15794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:40783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:34.932920 15794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:34.933164 15794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:34.937798 15800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:34.937809 15799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.937899 15794 server_base.cc:1034] running on GCE node
W20260925 05:05:34.937767 15802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:34.938194 15794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:34.938441 15794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:34.939591 15794 hybrid_clock.cc:648] HybridClock initialized: now 1790312734939589 us; error 32 us; skew 500 ppm
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:34.941948 15794 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:34.942384 15794 webserver.cc:460] Webserver started at http://127.14.33.2:35519/ using document root <none> and password file <none>
I20260925 05:05:34.942591 15794 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:34.942653 15794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:34.942782 15794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:34.943713 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "8abc3441fbce40509b8a6fdd1d3caaa9"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "69a6dad6186d988a60a9ce0c36382e31"
server_key_iv: "bca6c2757be70d3c8346a6d01671fe9f"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.944069 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "8abc3441fbce40509b8a6fdd1d3caaa9"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "69a6dad6186d988a60a9ce0c36382e31"
server_key_iv: "bca6c2757be70d3c8346a6d01671fe9f"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.945286 15794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:34.945926 15809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.946089 15794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.946185 15794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "8abc3441fbce40509b8a6fdd1d3caaa9"
format_stamp: "Formatted at 2026-09-25 05:05:34 on dist-test-slave-4dmq"
server_key: "69a6dad6186d988a60a9ce0c36382e31"
server_key_iv: "bca6c2757be70d3c8346a6d01671fe9f"
server_key_version: "encryptionkey@0"
I20260925 05:05:34.946264 15794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:34.967629 15794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:34.969288 15794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:34.969431 15794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:34.969959 15794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:34.970005 15794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:34.970034 15794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:34.970052 15794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:05:34 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:34.976557 15794 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:42359
I20260925 05:05:34.976624 15924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:42359 every 8 connection(s)
I20260925 05:05:34.976945 15794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:34.978941 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.970335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:37059 (local address 127.14.33.62:40783)
0925 05:05:34.970405 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.970408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.970557 (+ 149us) server_negotiation.cc:407] Connection header received
0925 05:05:34.970923 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.970924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.970937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.971008 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.971594 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.972357 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.973634 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.973853 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.973924 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.976454 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.976456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.976456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:34.976464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.977935 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.978256 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.978257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.978258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.978317 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.978502 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.978503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.978503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.978592 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.978635 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.978830 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.978884 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:05:34.979074 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.970313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.14.33.2:37059)
0925 05:05:34.970437 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:34.970470 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:34.970838 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:34.971068 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:34.971071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:34.971268 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:34.971524 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.971528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.972473 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.972474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.973522 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.973525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.973926 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.973927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.974082 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.974615 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:34.974623 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:34.976338 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:34.978001 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.978006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.978007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.978190 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.978371 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.978372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.978372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.978453 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.978640 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:34.978643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:34.978732 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 05:05:34.978805 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:05:34.978898 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260925 05:05:34.979508 15925 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40783
I20260925 05:05:34.979583 15925 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:34.979750 15925 heartbeater.cc:511] Master 127.14.33.62:40783 requested a full tablet report, sending...
I20260925 05:05:34.980054 15581 ts_manager.cc:194] Registered new tserver with Master: 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2:42359)
I20260925 05:05:34.980499 15581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:37059
I20260925 05:05:34.981536 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15794
I20260925 05:05:34.981629 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:34.981786 14468 external_mini_cluster.cc:1428] Setting key 438cf0fc3247b2a04a83e4261c12041b
I20260925 05:05:34.985508 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.980375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:38551 (local address 127.14.33.62:40783)
0925 05:05:34.980409 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:34.980411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:34.980480 (+ 69us) server_negotiation.cc:407] Connection header received
0925 05:05:34.980534 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:34.980535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:34.980541 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:34.980596 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:34.980946 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.981418 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.982200 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.982458 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.982521 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.983234 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:34.983235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:34.983236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:34.983242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:34.984483 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.984830 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.984831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.984831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.984873 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:34.985047 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:34.985048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:34.985048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:34.985211 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:34.985268 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:34.985408 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:05:34.985458 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:05:34.985630 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:34.980308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.14.33.2:38551)
0925 05:05:34.980423 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:34.980432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:34.980485 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:34.980615 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:34.980616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:34.980810 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:34.980866 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.980867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.981502 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.981503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.982141 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:34.982143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:34.982532 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:34.982533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:34.982582 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:34.982940 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:34.982943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:34.983159 (+ 216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:34.984552 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.984553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.984553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.984764 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.984933 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:34.984934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:34.984934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:34.984997 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:34.985277 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:34.985278 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:34.985321 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 05:05:34.985393 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:05:34.985494 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:35.003444 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40783
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:35.075745 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:35.075909 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:35.075930 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:35.077035 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:35.077073 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:35.077091 15934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:05:35.077148 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:35.078464 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:40783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:35.078704 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:35.078924 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:35.083585 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:35.083575 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:35.083591 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:35.083724 15934 server_base.cc:1034] running on GCE node
I20260925 05:05:35.084028 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:35.084270 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:35.085419 15934 hybrid_clock.cc:648] HybridClock initialized: now 1790312735085418 us; error 42 us; skew 500 ppm
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[15536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312735, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:35.087592 15934 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:35.087958 15934 webserver.cc:460] Webserver started at http://127.14.33.3:46217/ using document root <none> and password file <none>
I20260925 05:05:35.088151 15934 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:35.088212 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:35.088349 15934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:35.089282 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "718c8b732f4542d0b5ed7a6f1cc19055"
format_stamp: "Formatted at 2026-09-25 05:05:35 on dist-test-slave-4dmq"
server_key: "3a55a8878d8ce46f9da8f99be0bfa56f"
server_key_iv: "61e4ba976016ee728def6f5482142201"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.089588 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "718c8b732f4542d0b5ed7a6f1cc19055"
format_stamp: "Formatted at 2026-09-25 05:05:35 on dist-test-slave-4dmq"
server_key: "3a55a8878d8ce46f9da8f99be0bfa56f"
server_key_iv: "61e4ba976016ee728def6f5482142201"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.090780 15934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:35.091392 15949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.091542 15934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:35.091629 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "718c8b732f4542d0b5ed7a6f1cc19055"
format_stamp: "Formatted at 2026-09-25 05:05:35 on dist-test-slave-4dmq"
server_key: "3a55a8878d8ce46f9da8f99be0bfa56f"
server_key_iv: "61e4ba976016ee728def6f5482142201"
server_key_version: "encryptionkey@0"
I20260925 05:05:35.091699 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:35.106191 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:35.107728 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:35.107870 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:35.108337 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:35.108389 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.108426 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:35.108444 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312735, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:35.115355 15934 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:45237
I20260925 05:05:35.115413 16064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:45237 every 8 connection(s)
I20260925 05:05:35.115648 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:35.116547 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.108676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:40799 (local address 127.14.33.62:40783)
0925 05:05:35.108743 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.108746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.108981 (+ 235us) server_negotiation.cc:407] Connection header received
0925 05:05:35.109420 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.109422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.109437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.109518 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.110018 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.110492 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.111458 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.111650 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.111719 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.113863 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.113865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.113865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:35.113871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.115309 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.115664 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.115665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.115666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.115721 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.116008 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.116009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.116009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.116125 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.116181 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.116426 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.116490 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:05:35.116674 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.108674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.14.33.3:40799)
0925 05:05:35.108859 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.108894 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.109318 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.109558 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.109561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.109756 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:35.109911 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.109914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.110579 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.110580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.111377 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.111379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.111724 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.111725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.111868 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.112309 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:35.112316 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:35.113756 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:35.115393 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:35.115398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:35.115400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:35.115593 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:35.115834 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:35.115835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:35.115835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:35.115950 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:35.116185 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:35.116188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:35.116336 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.116407 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.116502 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":127,"threads_started":1}
I20260925 05:05:35.117168 16065 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40783
I20260925 05:05:35.117272 16065 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:35.117295 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 15934
I20260925 05:05:35.117398 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:35.117439 16065 heartbeater.cc:511] Master 127.14.33.62:40783 requested a full tablet report, sending...
I20260925 05:05:35.117569 14468 external_mini_cluster.cc:1428] Setting key 107f82ada7a6ce45b782d3b1ca958f45
I20260925 05:05:35.117811 15580 ts_manager.cc:194] Registered new tserver with Master: 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237)
I20260925 05:05:35.118264 15580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:40799
I20260925 05:05:35.118692 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:35.123525 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.118141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:41233 (local address 127.14.33.62:40783)
0925 05:05:35.118173 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.118175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.118313 (+ 138us) server_negotiation.cc:407] Connection header received
0925 05:05:35.118392 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.118393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.118402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.118429 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.118806 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.119284 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.119863 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.120148 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.120210 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.120989 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.120991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.120991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:35.120997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.122505 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.122835 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.122836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.122836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.122889 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.123098 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.123099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.123099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.123219 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.123266 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.123426 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.123474 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:05:35.123660 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.118104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.14.33.3:41233)
0925 05:05:35.118251 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.118261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.118323 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.118480 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.118481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.118665 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:35.118724 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.118726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.119370 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.119370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.119789 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.119790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.120234 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.120234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.120291 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.120669 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:35.120673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:35.120910 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:35.122583 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:35.122584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:35.122584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:35.122742 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:35.122952 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:35.122952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:35.122952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:35.123021 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:35.123288 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:35.123289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:35.123335 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.123412 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.123515 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[15536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:35.139523 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.133692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.14.33.62:40783)
0925 05:05:35.133755 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.133757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.133764 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:05:35.133839 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.133840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.133847 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.133888 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.134409 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.134893 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.135506 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.135657 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.135698 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.137586 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.137587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.137588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:35.137593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.138736 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.139030 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.139031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.139031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.139075 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.139189 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.139189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.139190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.139269 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.139344 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.139399 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.139446 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:05:35.140887 15580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:35.142383 15580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:35.147891 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.144320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45237 (local address 127.0.0.1:54338)
0925 05:05:35.144598 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.144608 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.144657 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.145035 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.145036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.145038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.145073 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.145074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.146236 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.146237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.146913 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.146915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.147704 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.147705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.147754 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.147764 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.147816 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.147847 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"spinlock_wait_cycles":768,"thread_start_us":68,"threads_started":1}
I20260925 05:05:35.147953 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.144455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.14.33.3:45237)
0925 05:05:35.144906 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.144909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.144917 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:35.144945 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.144947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.144959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.145024 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.145257 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.146157 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.147022 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.147630 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.147698 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.147754 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.147837 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.147866 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":72,"threads_started":1}
I20260925 05:05:35.148540 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.144320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45319 (local address 127.0.0.1:46622)
0925 05:05:35.144678 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.144686 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.144733 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.144873 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.144874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.144876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.144924 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.144925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.145948 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.145949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.147180 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.147183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.148300 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.148301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.148383 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.148388 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.148437 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.148474 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":33,"thread_start_us":57,"threads_started":1}
I20260925 05:05:35.148634 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.144472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.14.33.1:45319)
0925 05:05:35.144632 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.144636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.144731 (+ 95us) server_negotiation.cc:407] Connection header received
0925 05:05:35.144792 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.144795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.144808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.144876 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.145188 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.145835 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.147248 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.147895 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.147955 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.148005 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.148463 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.148532 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260925 05:05:35.148648 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 804d852204ba45e1a57ad0d5890bcd91 (DEFAULT_TABLE table=test-table [id=d5e90175f58a4e46bc8786b8e0795ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.148919 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.144318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:42359 (local address 127.0.0.1:45666)
0925 05:05:35.144454 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.144485 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.144545 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.144819 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.144822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.144824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.144880 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.144881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.146062 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.146062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.147180 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.147183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.148726 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.148727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.148783 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.148788 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.148818 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.148864 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":29,"thread_start_us":45,"threads_started":1}
I20260925 05:05:35.148989 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804d852204ba45e1a57ad0d5890bcd91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.149350 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.144524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.14.33.2:42359)
0925 05:05:35.144680 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.144683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.144690 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:05:35.144719 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.144720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.144730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.144814 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.144958 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.145985 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.148035 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.148617 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.149018 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.149073 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.149095 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.149150 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260925 05:05:35.149387 15719 tablet_service.cc:1467] Processing CreateTablet for tablet 804d852204ba45e1a57ad0d5890bcd91 (DEFAULT_TABLE table=test-table [id=d5e90175f58a4e46bc8786b8e0795ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.149727 15719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804d852204ba45e1a57ad0d5890bcd91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.151415 16085 tablet_bootstrap.cc:492] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Bootstrap starting.
I20260925 05:05:35.152246 16085 tablet_bootstrap.cc:654] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.152554 16085 log.cc:824] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:35.152781 15859 tablet_service.cc:1467] Processing CreateTablet for tablet 804d852204ba45e1a57ad0d5890bcd91 (DEFAULT_TABLE table=test-table [id=d5e90175f58a4e46bc8786b8e0795ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.153088 15859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804d852204ba45e1a57ad0d5890bcd91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.153251 16085 tablet_bootstrap.cc:492] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: No bootstrap required, opened a new log
I20260925 05:05:35.153309 16085 ts_tablet_manager.cc:1397] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:35.155228 16085 raft_consensus.cc:348] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.155390 16085 raft_consensus.cc:374] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.155423 16085 raft_consensus.cc:729] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718c8b732f4542d0b5ed7a6f1cc19055, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.155534 16085 consensus_queue.cc:260] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.155822 16065 heartbeater.cc:503] Master 127.14.33.62:40783 was elected leader, sending a full tablet report...
I20260925 05:05:35.156075 16085 ts_tablet_manager.cc:1428] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:35.156476 16089 tablet_bootstrap.cc:492] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Bootstrap starting.
I20260925 05:05:35.156594 16086 tablet_bootstrap.cc:492] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Bootstrap starting.
I20260925 05:05:35.157181 16086 tablet_bootstrap.cc:654] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.157193 16089 tablet_bootstrap.cc:654] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.157505 16089 log.cc:824] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:35.157503 16086 log.cc:824] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:35.158154 16086 tablet_bootstrap.cc:492] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: No bootstrap required, opened a new log
I20260925 05:05:35.158224 16086 ts_tablet_manager.cc:1397] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:05:35.158349 16089 tablet_bootstrap.cc:492] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: No bootstrap required, opened a new log
I20260925 05:05:35.158407 16089 ts_tablet_manager.cc:1397] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:05:35.160127 16086 raft_consensus.cc:348] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.160296 16086 raft_consensus.cc:374] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.160233 16089 raft_consensus.cc:348] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.160331 16086 raft_consensus.cc:729] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184fbd28dfa7400e90eb4596785a52b5, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.160363 16089 raft_consensus.cc:374] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.160400 16089 raft_consensus.cc:729] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abc3441fbce40509b8a6fdd1d3caaa9, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.160439 16086 consensus_queue.cc:260] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.160493 16089 consensus_queue.cc:260] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.160686 16086 ts_tablet_manager.cc:1428] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:05:35.160709 16089 ts_tablet_manager.cc:1428] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:05:35.160712 15785 heartbeater.cc:503] Master 127.14.33.62:40783 was elected leader, sending a full tablet report...
I20260925 05:05:35.160825 15925 heartbeater.cc:503] Master 127.14.33.62:40783 was elected leader, sending a full tablet report...
W20260925 05:05:35.227511 15926 tablet.cc:2369] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:35.330786 15786 tablet.cc:2369] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:35.366161 16066 tablet.cc:2369] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:35.484544 16088 raft_consensus.cc:482] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:35.484694 16088 raft_consensus.cc:504] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.485051 16088 leader_election.cc:288] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319), 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2:42359)
I20260925 05:05:35.487895 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.485335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:50385 (local address 127.14.33.2:42359)
0925 05:05:35.485377 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.485379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.485515 (+ 136us) server_negotiation.cc:407] Connection header received
0925 05:05:35.485558 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.485559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.485569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.485616 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.485779 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.486240 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.487179 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.487563 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.487612 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.487620 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.487790 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.487832 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:05:35.487920 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.485259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:42359 (local address 127.14.33.3:50385)
0925 05:05:35.485407 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.485425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.485507 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.485639 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.485639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.485641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.485714 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.485716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.486317 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.486318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.487101 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.487103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.487633 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.487634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.487723 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.487726 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.487771 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.487817 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":28,"thread_start_us":83,"threads_started":1}
I20260925 05:05:35.487977 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.485524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:33591 (local address 127.14.33.1:45319)
0925 05:05:35.485594 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.485596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.485620 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:05:35.485644 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.485645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.485655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.485717 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.485871 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.486325 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.487278 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.487631 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.487694 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.487704 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.487872 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.487908 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:05:35.488209 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804d852204ba45e1a57ad0d5890bcd91" candidate_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" is_pre_election: true
I20260925 05:05:35.488327 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.485259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45319 (local address 127.14.33.3:33591)
0925 05:05:35.485322 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.485334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.485394 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.485724 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.485725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.485726 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.485798 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.485800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.486401 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.486402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.487198 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.487200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.487709 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.487709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.487802 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.487805 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.487852 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.487891 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:05:35.488345 15879 raft_consensus.cc:2454] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718c8b732f4542d0b5ed7a6f1cc19055 in term 0.
I20260925 05:05:35.488595 15953 leader_election.cc:302] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718c8b732f4542d0b5ed7a6f1cc19055, 8abc3441fbce40509b8a6fdd1d3caaa9; no voters:
I20260925 05:05:35.488600 15739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804d852204ba45e1a57ad0d5890bcd91" candidate_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184fbd28dfa7400e90eb4596785a52b5" is_pre_election: true
I20260925 05:05:35.488719 15739 raft_consensus.cc:2454] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718c8b732f4542d0b5ed7a6f1cc19055 in term 0.
I20260925 05:05:35.488715 16088 raft_consensus.cc:2781] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:35.488771 16088 raft_consensus.cc:482] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:35.488811 16088 raft_consensus.cc:3037] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.489707 16088 raft_consensus.cc:504] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.489840 16088 leader_election.cc:288] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [CANDIDATE]: Term 1 election: Requested vote from peers 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319), 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2:42359)
I20260925 05:05:35.489950 15739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804d852204ba45e1a57ad0d5890bcd91" candidate_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184fbd28dfa7400e90eb4596785a52b5"
I20260925 05:05:35.489967 15879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804d852204ba45e1a57ad0d5890bcd91" candidate_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9"
I20260925 05:05:35.490031 15879 raft_consensus.cc:3037] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.490041 15739 raft_consensus.cc:3037] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.490666 15879 raft_consensus.cc:2454] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718c8b732f4542d0b5ed7a6f1cc19055 in term 1.
I20260925 05:05:35.490672 15739 raft_consensus.cc:2454] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718c8b732f4542d0b5ed7a6f1cc19055 in term 1.
I20260925 05:05:35.490826 15950 leader_election.cc:302] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184fbd28dfa7400e90eb4596785a52b5, 718c8b732f4542d0b5ed7a6f1cc19055; no voters:
I20260925 05:05:35.490936 16088 raft_consensus.cc:2781] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:35.491080 16088 raft_consensus.cc:686] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 LEADER]: Becoming Leader. State: Replica: 718c8b732f4542d0b5ed7a6f1cc19055, State: Running, Role: LEADER
I20260925 05:05:35.491161 16088 consensus_queue.cc:237] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } }
I20260925 05:05:35.492022 15580 catalog_manager.cc:5495] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:35.498651 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.496667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.14.33.3:45237)
0925 05:05:35.496699 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.496701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.496711 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:05:35.496794 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.496794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.496806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.496859 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:35.496976 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.497420 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.498054 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.498228 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.498271 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.498419 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.498492 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.498542 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.498584 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.498597 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:05:35.499977 15879 raft_consensus.cc:1261] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Refusing update from remote peer 718c8b732f4542d0b5ed7a6f1cc19055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.499977 15739 raft_consensus.cc:1261] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Refusing update from remote peer 718c8b732f4542d0b5ed7a6f1cc19055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.500238 16088 consensus_queue.cc:1035] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.500368 16095 consensus_queue.cc:1035] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.502213 16098 mvcc.cc:205] Tried to move back new op lower bound from 7333120964605956096 to 7333120964572323840. Current Snapshot: MvccSnapshot[applied={T|T < 7333120964605956096}]
I20260925 05:05:35.502641 16099 mvcc.cc:205] Tried to move back new op lower bound from 7333120964605956096 to 7333120964572323840. Current Snapshot: MvccSnapshot[applied={T|T < 7333120964605956096}]
I20260925 05:05:35.505364 15581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:35.505443 15581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:35.506072 15581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: Sending DeleteTablet for 3 replicas of tablet 804d852204ba45e1a57ad0d5890bcd91
I20260925 05:05:35.506351 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 804d852204ba45e1a57ad0d5890bcd91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260925 05:05:35.506369 15719 tablet_service.cc:1514] Processing DeleteTablet for tablet 804d852204ba45e1a57ad0d5890bcd91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46622
I20260925 05:05:35.506368 15859 tablet_service.cc:1514] Processing DeleteTablet for tablet 804d852204ba45e1a57ad0d5890bcd91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260925 05:05:35.506522 16108 tablet_replica.cc:330] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: stopping tablet replica
I20260925 05:05:35.506604 16108 raft_consensus.cc:2229] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:35.506675 16108 raft_consensus.cc:2258] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:35.506800 16109 tablet_replica.cc:330] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: stopping tablet replica
I20260925 05:05:35.506903 16109 raft_consensus.cc:2229] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:35.506870 15581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:05:35.506985 16109 raft_consensus.cc:2258] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 05:05:35.507009 15581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:35.507289 16108 ts_tablet_manager.cc:1905] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.507462 16109 ts_tablet_manager.cc:1905] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.508337 15719 tablet_service.cc:1467] Processing CreateTablet for tablet 83d8cb5134f54f0994ef56a21874be7d (DEFAULT_TABLE table=test-table [id=8c5eaa21e8664e76b7da842073146721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.508376 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 83d8cb5134f54f0994ef56a21874be7d (DEFAULT_TABLE table=test-table [id=8c5eaa21e8664e76b7da842073146721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.508440 15719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d8cb5134f54f0994ef56a21874be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.508474 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d8cb5134f54f0994ef56a21874be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.508965 15859 tablet_service.cc:1467] Processing CreateTablet for tablet 83d8cb5134f54f0994ef56a21874be7d (DEFAULT_TABLE table=test-table [id=8c5eaa21e8664e76b7da842073146721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:35.509063 15859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d8cb5134f54f0994ef56a21874be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.509137 16108 ts_tablet_manager.cc:1918] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:35.509215 16108 log.cc:1196] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/804d852204ba45e1a57ad0d5890bcd91
I20260925 05:05:35.509462 16108 ts_tablet_manager.cc:1939] T 804d852204ba45e1a57ad0d5890bcd91 P 184fbd28dfa7400e90eb4596785a52b5: Deleting consensus metadata
I20260925 05:05:35.509750 16086 tablet_bootstrap.cc:492] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Bootstrap starting.
I20260925 05:05:35.510075 16086 tablet_bootstrap.cc:654] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.510299 16089 tablet_bootstrap.cc:492] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Bootstrap starting.
I20260925 05:05:35.510605 15565 catalog_manager.cc:4841] TS 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319): tablet 804d852204ba45e1a57ad0d5890bcd91 (table test-table [id=d5e90175f58a4e46bc8786b8e0795ebc]) successfully deleted
I20260925 05:05:35.510645 16089 tablet_bootstrap.cc:654] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.510749 16086 tablet_bootstrap.cc:492] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: No bootstrap required, opened a new log
I20260925 05:05:35.510793 16086 ts_tablet_manager.cc:1397] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:35.510918 16086 raft_consensus.cc:348] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.510988 16086 raft_consensus.cc:374] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.511013 16086 raft_consensus.cc:729] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184fbd28dfa7400e90eb4596785a52b5, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.511066 16086 consensus_queue.cc:260] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.511173 16086 ts_tablet_manager.cc:1428] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:35.512041 16085 tablet_bootstrap.cc:492] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Bootstrap starting.
I20260925 05:05:35.512243 16089 tablet_bootstrap.cc:492] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: No bootstrap required, opened a new log
I20260925 05:05:35.512300 16089 ts_tablet_manager.cc:1397] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:05:35.512434 16085 tablet_bootstrap.cc:654] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.512439 16089 raft_consensus.cc:348] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.512493 16089 raft_consensus.cc:374] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.512517 16089 raft_consensus.cc:729] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abc3441fbce40509b8a6fdd1d3caaa9, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.512552 16089 consensus_queue.cc:260] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.512648 16089 ts_tablet_manager.cc:1428] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.512979 16085 tablet_bootstrap.cc:492] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: No bootstrap required, opened a new log
I20260925 05:05:35.513023 16085 ts_tablet_manager.cc:1397] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:35.513193 16085 raft_consensus.cc:348] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.513279 16085 raft_consensus.cc:374] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.513309 16085 raft_consensus.cc:729] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718c8b732f4542d0b5ed7a6f1cc19055, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.513262 16110 tablet_replica.cc:330] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: stopping tablet replica
I20260925 05:05:35.513350 16085 consensus_queue.cc:260] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.513433 16085 ts_tablet_manager.cc:1428] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:35.513526 16110 raft_consensus.cc:2229] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:35.513662 16110 raft_consensus.cc:2258] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:35.514070 16110 ts_tablet_manager.cc:1905] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.514510 16109 ts_tablet_manager.cc:1918] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:35.514580 16109 log.cc:1196] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/804d852204ba45e1a57ad0d5890bcd91
I20260925 05:05:35.514814 16109 ts_tablet_manager.cc:1939] T 804d852204ba45e1a57ad0d5890bcd91 P 718c8b732f4542d0b5ed7a6f1cc19055: Deleting consensus metadata
I20260925 05:05:35.515385 15566 catalog_manager.cc:4841] TS 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237): tablet 804d852204ba45e1a57ad0d5890bcd91 (table test-table [id=d5e90175f58a4e46bc8786b8e0795ebc]) successfully deleted
I20260925 05:05:35.515699 16110 ts_tablet_manager.cc:1918] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:35.515784 16110 log.cc:1196] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/804d852204ba45e1a57ad0d5890bcd91
I20260925 05:05:35.516028 16110 ts_tablet_manager.cc:1939] T 804d852204ba45e1a57ad0d5890bcd91 P 8abc3441fbce40509b8a6fdd1d3caaa9: Deleting consensus metadata
I20260925 05:05:35.516393 15568 catalog_manager.cc:4841] TS 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2:42359): tablet 804d852204ba45e1a57ad0d5890bcd91 (table test-table [id=d5e90175f58a4e46bc8786b8e0795ebc]) successfully deleted
I20260925 05:05:35.578171 16093 raft_consensus.cc:482] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:35.578300 16093 raft_consensus.cc:504] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.578647 16093 leader_election.cc:288] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319), 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237)
I20260925 05:05:35.581925 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.578872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:53605 (local address 127.14.33.1:45319)
0925 05:05:35.578921 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.578922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.579022 (+ 100us) server_negotiation.cc:407] Connection header received
0925 05:05:35.579104 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.579106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.579115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.579167 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.579330 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.579902 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.581256 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.581588 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.581654 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.581662 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.581829 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.581862 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:05:35.581948 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.578814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45319 (local address 127.14.33.2:53605)
0925 05:05:35.578966 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.578977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.579033 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.579178 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.579179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.579181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.579255 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.579257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.579995 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.579996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.581134 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.581136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.581707 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.581708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.581755 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.581758 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.581822 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.581860 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260925 05:05:35.582129 15739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d8cb5134f54f0994ef56a21874be7d" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184fbd28dfa7400e90eb4596785a52b5" is_pre_election: true
I20260925 05:05:35.582196 15739 raft_consensus.cc:2454] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 0.
I20260925 05:05:35.582199 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.578973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:32981 (local address 127.14.33.3:45237)
0925 05:05:35.579005 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.579007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.579112 (+ 105us) server_negotiation.cc:407] Connection header received
0925 05:05:35.579225 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.579226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.579241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.579264 (+ 23us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:35.579487 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.580120 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.581252 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.581763 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.581841 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.581850 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.582104 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.582137 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:05:35.582381 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.578839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45237 (local address 127.14.33.2:32981)
0925 05:05:35.579053 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.579062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.579118 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.579333 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.579333 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.579335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:35.579398 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.579400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.580200 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.580201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.581134 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.581136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.581900 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.581901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.581967 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.581970 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.582272 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.582312 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260925 05:05:35.582399 15810 leader_election.cc:302] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184fbd28dfa7400e90eb4596785a52b5, 8abc3441fbce40509b8a6fdd1d3caaa9; no voters:
I20260925 05:05:35.582537 16093 raft_consensus.cc:2781] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:35.582574 16093 raft_consensus.cc:482] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:35.582593 16093 raft_consensus.cc:3037] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.582650 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d8cb5134f54f0994ef56a21874be7d" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" is_pre_election: true
I20260925 05:05:35.582758 16019 raft_consensus.cc:2454] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 0.
I20260925 05:05:35.583413 16093 raft_consensus.cc:504] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.583535 16093 leader_election.cc:288] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 election: Requested vote from peers 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319), 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237)
I20260925 05:05:35.583647 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d8cb5134f54f0994ef56a21874be7d" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718c8b732f4542d0b5ed7a6f1cc19055"
I20260925 05:05:35.583714 16019 raft_consensus.cc:3037] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.583714 15739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d8cb5134f54f0994ef56a21874be7d" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184fbd28dfa7400e90eb4596785a52b5"
I20260925 05:05:35.583765 15739 raft_consensus.cc:3037] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.584549 16019 raft_consensus.cc:2454] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 1.
I20260925 05:05:35.584717 15811 leader_election.cc:302] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718c8b732f4542d0b5ed7a6f1cc19055, 8abc3441fbce40509b8a6fdd1d3caaa9; no voters:
I20260925 05:05:35.584801 16093 raft_consensus.cc:2781] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:35.584861 15739 raft_consensus.cc:2454] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 1.
I20260925 05:05:35.584930 16093 raft_consensus.cc:686] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 LEADER]: Becoming Leader. State: Replica: 8abc3441fbce40509b8a6fdd1d3caaa9, State: Running, Role: LEADER
I20260925 05:05:35.585000 16093 consensus_queue.cc:237] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.586000 15581 catalog_manager.cc:5495] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2). New cstate: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312734, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:35.627601 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.621695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40783 (local address 127.0.0.1:41696)
0925 05:05:35.621862 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.621871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.621928 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.622043 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.622044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.622329 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:35.622374 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.622376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.622973 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.622973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.623469 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.623470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.623773 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.623774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.623818 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.624326 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:35.624330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:35.625556 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:35.626787 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:35.626792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:35.626794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:35.626961 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:35.627121 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:35.627121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:35.627122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:35.627169 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:35.627360 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:35.627365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:35.627399 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.627447 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.627522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260925 05:05:35.627647 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.621736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.14.33.62:40783)
0925 05:05:35.621796 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.621798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.621930 (+ 132us) server_negotiation.cc:407] Connection header received
0925 05:05:35.621978 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.621978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.621987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.622024 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.622435 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.622909 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.623532 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.623701 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.623747 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.625639 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.625640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.625640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:35.625646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.626722 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.627025 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.627026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.627026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.627069 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.627225 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.627226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.627226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.627311 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.627355 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.627475 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.627562 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":36}
I20260925 05:05:35.629310 15581 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:41696:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:05:35.629496 15581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:35.630884 15859 tablet_service.cc:1467] Processing CreateTablet for tablet 7701f36a0f9e4c3d921be67b0b4c773a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=701d0b38a0b74e2d9b53b05d6f8f9fd3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:05:35.630897 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 7701f36a0f9e4c3d921be67b0b4c773a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=701d0b38a0b74e2d9b53b05d6f8f9fd3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:05:35.630897 15719 tablet_service.cc:1467] Processing CreateTablet for tablet 7701f36a0f9e4c3d921be67b0b4c773a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=701d0b38a0b74e2d9b53b05d6f8f9fd3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:05:35.631033 15719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7701f36a0f9e4c3d921be67b0b4c773a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.631035 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7701f36a0f9e4c3d921be67b0b4c773a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.631033 15859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7701f36a0f9e4c3d921be67b0b4c773a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:35.632136 16089 tablet_bootstrap.cc:492] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: Bootstrap starting.
I20260925 05:05:35.632426 16085 tablet_bootstrap.cc:492] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: Bootstrap starting.
I20260925 05:05:35.632457 16086 tablet_bootstrap.cc:492] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: Bootstrap starting.
I20260925 05:05:35.632556 16089 tablet_bootstrap.cc:654] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.632732 16085 tablet_bootstrap.cc:654] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.632921 16086 tablet_bootstrap.cc:654] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:35.633315 16089 tablet_bootstrap.cc:492] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: No bootstrap required, opened a new log
I20260925 05:05:35.633383 16089 ts_tablet_manager.cc:1397] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:35.633533 16086 tablet_bootstrap.cc:492] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: No bootstrap required, opened a new log
I20260925 05:05:35.633530 16089 raft_consensus.cc:348] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.633587 16086 ts_tablet_manager.cc:1397] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:35.633597 16089 raft_consensus.cc:374] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.633621 16089 raft_consensus.cc:729] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abc3441fbce40509b8a6fdd1d3caaa9, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.633669 16089 consensus_queue.cc:260] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.633749 16089 ts_tablet_manager.cc:1428] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.633751 16085 tablet_bootstrap.cc:492] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: No bootstrap required, opened a new log
I20260925 05:05:35.633787 16085 ts_tablet_manager.cc:1397] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:35.633728 16086 raft_consensus.cc:348] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.633818 16086 raft_consensus.cc:374] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.633847 16086 raft_consensus.cc:729] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184fbd28dfa7400e90eb4596785a52b5, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.633888 16086 consensus_queue.cc:260] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.633886 16085 raft_consensus.cc:348] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.633940 16085 raft_consensus.cc:374] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:35.633965 16085 raft_consensus.cc:729] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718c8b732f4542d0b5ed7a6f1cc19055, State: Initialized, Role: FOLLOWER
I20260925 05:05:35.633980 16086 ts_tablet_manager.cc:1428] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.634001 16085 consensus_queue.cc:260] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.634085 16085 ts_tablet_manager.cc:1428] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:35.634243 16093 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.634342 16093 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.634485 16092 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.634600 16092 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.634801 16103 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.634884 16103 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.748188 16093 raft_consensus.cc:482] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:35.748287 16093 raft_consensus.cc:504] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.748478 16093 leader_election.cc:288] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319), 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237)
I20260925 05:05:35.748718 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7701f36a0f9e4c3d921be67b0b4c773a" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" is_pre_election: true
I20260925 05:05:35.748745 15739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7701f36a0f9e4c3d921be67b0b4c773a" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184fbd28dfa7400e90eb4596785a52b5" is_pre_election: true
I20260925 05:05:35.748822 15739 raft_consensus.cc:2454] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 0.
I20260925 05:05:35.748822 16019 raft_consensus.cc:2454] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 0.
I20260925 05:05:35.748955 15810 leader_election.cc:302] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184fbd28dfa7400e90eb4596785a52b5, 8abc3441fbce40509b8a6fdd1d3caaa9; no voters:
I20260925 05:05:35.749085 16093 raft_consensus.cc:2781] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:35.749179 16093 raft_consensus.cc:482] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:35.749228 16093 raft_consensus.cc:3037] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.749872 16093 raft_consensus.cc:504] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.750044 16093 leader_election.cc:288] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 election: Requested vote from peers 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319), 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237)
I20260925 05:05:35.750195 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7701f36a0f9e4c3d921be67b0b4c773a" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718c8b732f4542d0b5ed7a6f1cc19055"
I20260925 05:05:35.750283 16019 raft_consensus.cc:3037] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.750213 15739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7701f36a0f9e4c3d921be67b0b4c773a" candidate_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184fbd28dfa7400e90eb4596785a52b5"
I20260925 05:05:35.750324 15739 raft_consensus.cc:3037] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:35.750824 16019 raft_consensus.cc:2454] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 1.
I20260925 05:05:35.750932 15739 raft_consensus.cc:2454] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8abc3441fbce40509b8a6fdd1d3caaa9 in term 1.
I20260925 05:05:35.750967 15811 leader_election.cc:302] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718c8b732f4542d0b5ed7a6f1cc19055, 8abc3441fbce40509b8a6fdd1d3caaa9; no voters:
I20260925 05:05:35.751057 16093 raft_consensus.cc:2781] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:35.751109 16093 raft_consensus.cc:686] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 LEADER]: Becoming Leader. State: Replica: 8abc3441fbce40509b8a6fdd1d3caaa9, State: Running, Role: LEADER
I20260925 05:05:35.751158 16093 consensus_queue.cc:237] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } }
I20260925 05:05:35.751420 16093 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: TxnStatusTablet state changed. Reason: New leader 8abc3441fbce40509b8a6fdd1d3caaa9. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.751483 16093 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:05:35.751658 16128 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:05:35.751757 16128 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:05:35.751701 15581 catalog_manager.cc:5495] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2). New cstate: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:35.778422 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.775700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:42359 (local address 127.0.0.1:45678)
0925 05:05:35.775820 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.775830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.775874 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.776029 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.776030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.776031 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:35.776072 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.776074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.776938 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.776938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.777492 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.777494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.777939 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.777939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.777982 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.777994 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.778284 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.778289 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.778331 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.778368 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:05:35.778424 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.775754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.14.33.2:42359)
0925 05:05:35.775804 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.775806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.775866 (+ 60us) server_negotiation.cc:407] Connection header received
0925 05:05:35.775935 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.775936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.775953 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.776019 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:35.776136 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.776865 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.777568 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.777867 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.777935 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.778066 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.778182 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.778288 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.778335 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.778357 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:05:35.779466 15739 raft_consensus.cc:1261] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Refusing update from remote peer 8abc3441fbce40509b8a6fdd1d3caaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.779557 16019 raft_consensus.cc:1261] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Refusing update from remote peer 8abc3441fbce40509b8a6fdd1d3caaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.779706 16093 consensus_queue.cc:1035] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.780007 16116 consensus_queue.cc:1035] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.780544 16092 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: TxnStatusTablet state changed. Reason: New leader 8abc3441fbce40509b8a6fdd1d3caaa9. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.780643 16092 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.781032 16103 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: TxnStatusTablet state changed. Reason: New leader 8abc3441fbce40509b8a6fdd1d3caaa9. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.781138 16103 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.781533 16093 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.781599 16093 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:05:35.781917 16100 mvcc.cc:205] Tried to move back new op lower bound from 7333120965750878208 to 7333120965637095424. Current Snapshot: MvccSnapshot[applied={T|T < 7333120965750878208}]
I20260925 05:05:35.782261 16092 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.782334 16092 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 184fbd28dfa7400e90eb4596785a52b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.782986 16116 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.783061 16116 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 8abc3441fbce40509b8a6fdd1d3caaa9: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:05:35.783689 16103 tablet_replica.cc:439] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 } } peers { permanent_uuid: "8abc3441fbce40509b8a6fdd1d3caaa9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 42359 } } peers { permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 } } }
I20260925 05:05:35.783766 16103 tablet_replica.cc:442] T 7701f36a0f9e4c3d921be67b0b4c773a P 718c8b732f4542d0b5ed7a6f1cc19055: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:35.785622 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.783300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.14.33.2:42359)
0925 05:05:35.783406 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.783408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.783416 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:35.783435 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.783436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.783444 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.783498 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:35.783630 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.784249 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.784934 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.785187 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.785258 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.785410 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.785439 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.785480 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.785541 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.785555 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 05:05:35.790452 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.787910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:44921 (local address 127.14.33.2:42359)
0925 05:05:35.787970 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.787972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.788054 (+ 82us) server_negotiation.cc:407] Connection header received
0925 05:05:35.788114 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.788114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.788120 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.788155 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:35.788268 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.788634 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.789650 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.789955 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.790045 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.790234 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.790259 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.790303 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.790377 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.790392 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:05:35.790552 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.787830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:42359 (local address 127.14.33.2:44921)
0925 05:05:35.787971 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:35.787981 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:35.788050 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:35.788165 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:35.788166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:35.788167 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:35.788220 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.788221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.788696 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.788697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.789584 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:35.789586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.790062 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.790063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:35.790138 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.790152 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:35.790321 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:35.790323 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:35.790376 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:05:35.790417 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260925 05:05:35.792657 16019 raft_consensus.cc:1261] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Refusing update from remote peer 8abc3441fbce40509b8a6fdd1d3caaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.792685 15739 raft_consensus.cc:1261] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Refusing update from remote peer 8abc3441fbce40509b8a6fdd1d3caaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:35.792869 16136 consensus_queue.cc:1035] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184fbd28dfa7400e90eb4596785a52b5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.792991 16116 consensus_queue.cc:1035] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718c8b732f4542d0b5ed7a6f1cc19055" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:35.803848 15581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:35.803970 15581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:35.804642 15581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b93fb05b8c214570b4a738127e004d0e: Sending DeleteTablet for 3 replicas of tablet 83d8cb5134f54f0994ef56a21874be7d
I20260925 05:05:35.804877 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d8cb5134f54f0994ef56a21874be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260925 05:05:35.804946 15719 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d8cb5134f54f0994ef56a21874be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46622
I20260925 05:05:35.804972 15859 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d8cb5134f54f0994ef56a21874be7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:35 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260925 05:05:35.805037 16108 tablet_replica.cc:330] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: stopping tablet replica
I20260925 05:05:35.805049 16109 tablet_replica.cc:330] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: stopping tablet replica
I20260925 05:05:35.805079 16108 raft_consensus.cc:2229] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:35.805095 16109 raft_consensus.cc:2229] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:35.805131 16108 pending_rounds.cc:62] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Trying to abort 2 pending ops.
I20260925 05:05:35.805184 16110 tablet_replica.cc:330] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: stopping tablet replica
I20260925 05:05:35.805251 16110 raft_consensus.cc:2229] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:35.805194 16108 pending_rounds.cc:68] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333120965805400064 op_type: PARTICIPANT_OP participant_request { tablet_id: "83d8cb5134f54f0994ef56a21874be7d" op { type: BEGIN_TXN txn_id: 0 } }
I20260925 05:05:35.805338 16108 pending_rounds.cc:68] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333120964957577216 op_type: NO_OP noop_request { }
I20260925 05:05:35.805352 16110 raft_consensus.cc:2258] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:35.805378 16108 raft_consensus.cc:2866] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260925 05:05:35.805413 16108 raft_consensus.cc:2258] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 05:05:35.805493 16138 consensus_queue.cc:1159] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "184fbd28dfa7400e90eb4596785a52b5" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260925 05:05:35.805531 16109 raft_consensus.cc:2258] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:35.805915 16109 ts_tablet_manager.cc:1905] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.806061 16110 ts_tablet_manager.cc:1905] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.807780 16110 ts_tablet_manager.cc:1918] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:35.807801 16108 ts_tablet_manager.cc:1905] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:35.807863 16110 log.cc:1196] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/83d8cb5134f54f0994ef56a21874be7d
I20260925 05:05:35.808102 16110 ts_tablet_manager.cc:1939] T 83d8cb5134f54f0994ef56a21874be7d P 8abc3441fbce40509b8a6fdd1d3caaa9: Deleting consensus metadata
I20260925 05:05:35.808385 15568 catalog_manager.cc:4841] TS 8abc3441fbce40509b8a6fdd1d3caaa9 (127.14.33.2:42359): tablet 83d8cb5134f54f0994ef56a21874be7d (table test-table [id=8c5eaa21e8664e76b7da842073146721]) successfully deleted
I20260925 05:05:35.809376 16108 ts_tablet_manager.cc:1918] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:35.809430 16108 log.cc:1196] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/83d8cb5134f54f0994ef56a21874be7d
I20260925 05:05:35.809644 16108 ts_tablet_manager.cc:1939] T 83d8cb5134f54f0994ef56a21874be7d P 184fbd28dfa7400e90eb4596785a52b5: Deleting consensus metadata
I20260925 05:05:35.809922 16109 ts_tablet_manager.cc:1918] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:35.809940 15565 catalog_manager.cc:4841] TS 184fbd28dfa7400e90eb4596785a52b5 (127.14.33.1:45319): tablet 83d8cb5134f54f0994ef56a21874be7d (table test-table [id=8c5eaa21e8664e76b7da842073146721]) successfully deleted
I20260925 05:05:35.809970 16109 log.cc:1196] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/83d8cb5134f54f0994ef56a21874be7d
I20260925 05:05:35.810171 16109 ts_tablet_manager.cc:1939] T 83d8cb5134f54f0994ef56a21874be7d P 718c8b732f4542d0b5ed7a6f1cc19055: Deleting consensus metadata
I20260925 05:05:35.810428 15566 catalog_manager.cc:4841] TS 718c8b732f4542d0b5ed7a6f1cc19055 (127.14.33.3:45237): tablet 83d8cb5134f54f0994ef56a21874be7d (table test-table [id=8c5eaa21e8664e76b7da842073146721]) successfully deleted
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[15536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:35.814896 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.805746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.14.33.1:45319)
0925 05:05:35.805789 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.805791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.806027 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:05:35.806123 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.806124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.806137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.806166 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.806921 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.807601 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.808654 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.808895 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.808970 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.812054 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.812063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.812063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:35.812070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.813762 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.814124 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.814127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.814127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.814202 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.814401 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.814402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.814403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.814533 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.814696 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.814734 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.814803 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:05:35.815250 15779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46632
I20260925 05:05:35.821945 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:35.816123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.14.33.62:40783)
0925 05:05:35.816168 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:35.816170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:35.816200 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:05:35.816266 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:35.816266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:35.816274 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:35.816319 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:35.816909 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.817396 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.818250 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:35.818413 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:35.818468 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:35.819638 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:35.819639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:35.819639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:35.819654 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:35.820948 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.821314 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.821314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.821315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.821363 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:35.821557 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:35.821557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:35.821558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:35.821649 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:35.821715 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:35.821842 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 05:05:35.821896 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 05:05:35.822186 15581 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 05:05:35.822906 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15654
I20260925 05:05:35.828065 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15794
I20260925 05:05:35.833037 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15934
I20260925 05:05:35.838006 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 15549
2026-09-25T05:05:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2378 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[16147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[16147](info): set up 2 sockets
Sep 25 05:05:35 dist-test-slave-4dmq krb5kdc[16147](info): commencing operation
krb5kdc: starting...
W20260925 05:05:36.849996 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:36 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:36.958235 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:46657
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:37.037359 16160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:37.037569 16160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:37.037601 16160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:37.038729 16160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:37.038792 16160 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:05:37.038813 16160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:37.038836 16160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:37.038865 16160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:37.038889 16160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:37.040258 16160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:46657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:37.040530 16160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:37.040741 16160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:37.045362 16168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.045392 16160 server_base.cc:1034] running on GCE node
W20260925 05:05:37.045406 16166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.045429 16165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.046042 16160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:37.046342 16160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:37.047505 16160 hybrid_clock.cc:648] HybridClock initialized: now 1790312737047495 us; error 43 us; skew 500 ppm
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:37.049779 16160 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:37.050098 16160 webserver.cc:460] Webserver started at http://127.14.33.62:32839/ using document root <none> and password file <none>
I20260925 05:05:37.050261 16160 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:37.050297 16160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:37.050371 16160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:37.051292 16160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "2b572b873be24c4b8cd8bac568a80fb8"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "179ed5c10a9038dd24bcf48e513e8304"
server_key_iv: "9c621218afe7ef8740f7adb06fbe9c86"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.051637 16160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "2b572b873be24c4b8cd8bac568a80fb8"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "179ed5c10a9038dd24bcf48e513e8304"
server_key_iv: "9c621218afe7ef8740f7adb06fbe9c86"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.052855 16160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:05:37.053568 16175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.053725 16160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.053803 16160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "2b572b873be24c4b8cd8bac568a80fb8"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "179ed5c10a9038dd24bcf48e513e8304"
server_key_iv: "9c621218afe7ef8740f7adb06fbe9c86"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.053853 16160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:37.071336 16160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:37.073391 16160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:37.073489 16160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:37.077507 16160 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:46657
I20260925 05:05:37.077586 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:46657 every 8 connection(s)
I20260925 05:05:37.077881 16160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:37.078517 16238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.080650 16238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Bootstrap starting.
I20260925 05:05:37.081300 16238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.081578 16238 log.cc:824] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:37.082203 16238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: No bootstrap required, opened a new log
I20260925 05:05:37.082439 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 16160
I20260925 05:05:37.082517 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:37.082685 14468 external_mini_cluster.cc:1428] Setting key 3db4ffeb20ba12f70e96dea47b14a92e
I20260925 05:05:37.084707 16238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b572b873be24c4b8cd8bac568a80fb8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46657 } }
I20260925 05:05:37.084870 16238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.084906 16238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b572b873be24c4b8cd8bac568a80fb8, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.085089 16238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b572b873be24c4b8cd8bac568a80fb8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46657 } }
I20260925 05:05:37.085207 16238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:37.085266 16238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:37.085317 16238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:37.085920 16238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b572b873be24c4b8cd8bac568a80fb8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46657 } }
I20260925 05:05:37.086018 16238 leader_election.cc:302] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b572b873be24c4b8cd8bac568a80fb8; no voters:
I20260925 05:05:37.086158 16238 leader_election.cc:288] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:37.086220 16243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:37.086369 16238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:37.086374 16243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 2b572b873be24c4b8cd8bac568a80fb8, State: Running, Role: LEADER
I20260925 05:05:37.086498 16243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b572b873be24c4b8cd8bac568a80fb8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46657 } }
I20260925 05:05:37.086891 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b572b873be24c4b8cd8bac568a80fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b572b873be24c4b8cd8bac568a80fb8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46657 } } }
I20260925 05:05:37.087037 16245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b572b873be24c4b8cd8bac568a80fb8. Latest consensus state: current_term: 1 leader_uuid: "2b572b873be24c4b8cd8bac568a80fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b572b873be24c4b8cd8bac568a80fb8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46657 } } }
I20260925 05:05:37.087318 16245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:37.087100 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:37.087563 16256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:37.088450 16256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:37.090085 16256 catalog_manager.cc:1347] Generated new cluster ID: 5507f856084644fe8af96d5b346a4e11
I20260925 05:05:37.090135 16256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:37.094206 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.083155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.14.33.62:46657)
0925 05:05:37.083333 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.083342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.083374 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:05:37.084677 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.084685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.084746 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.084901 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.085610 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.086943 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.088062 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.088452 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.088648 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.091089 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.091093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.091094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.091104 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.092687 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.093053 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.093056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.093056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.093228 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.093426 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.093427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.093428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.093649 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.093823 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.093895 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.094030 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260925 05:05:37.110316 16256 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:37.110819 16256 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:37.113415 16256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Generated new TSK 0
I20260925 05:05:37.113579 16256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:37.139907 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46657
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:37.243374 16264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:37.243573 16264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:37.243611 16264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:37.244722 16264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:37.244800 16264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:37.244850 16264 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:05:37.244911 16264 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:05:37.244951 16264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:37.246384 16264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:46657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:37.246619 16264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:37.246829 16264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:37.252290 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.252300 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.252416 16264 server_base.cc:1034] running on GCE node
W20260925 05:05:37.252313 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.252676 16264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:37.252945 16264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:37.254105 16264 hybrid_clock.cc:648] HybridClock initialized: now 1790312737254075 us; error 43 us; skew 500 ppm
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:37.256443 16264 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:37.256857 16264 webserver.cc:460] Webserver started at http://127.14.33.1:41213/ using document root <none> and password file <none>
I20260925 05:05:37.257076 16264 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:37.257158 16264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:37.257295 16264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:37.258284 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "b24f728225bb4a7e9494d4ecca09fec3"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "287e65fd165cdd958eea6a6b5191bab9"
server_key_iv: "f116a83d1c2b5fa712b1eb54fa6e480c"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.258644 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "b24f728225bb4a7e9494d4ecca09fec3"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "287e65fd165cdd958eea6a6b5191bab9"
server_key_iv: "f116a83d1c2b5fa712b1eb54fa6e480c"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.259915 16264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:37.260562 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.260727 16264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:37.260833 16264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "b24f728225bb4a7e9494d4ecca09fec3"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "287e65fd165cdd958eea6a6b5191bab9"
server_key_iv: "f116a83d1c2b5fa712b1eb54fa6e480c"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.260901 16264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:37.273360 16264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:37.275161 16264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:37.275305 16264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:37.275767 16264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:37.275808 16264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.275830 16264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:37.275841 16264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.282491 16264 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:32863
I20260925 05:05:37.282567 16394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:32863 every 8 connection(s)
I20260925 05:05:37.282837 16264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:37.284478 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 16264
I20260925 05:05:37.284591 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:37.284816 14468 external_mini_cluster.cc:1428] Setting key 02544fd73c76f7bfa4c040417bbb9093
I20260925 05:05:37.286092 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.276237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.14.33.1:47865)
0925 05:05:37.276405 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.276442 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.276893 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.277230 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.277233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.277587 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.277867 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.277873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.278913 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.278915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.280309 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.280313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.280848 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.280849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.280950 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.281544 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.281555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.283317 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.284671 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.284677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.284679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.284951 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.285198 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.285198 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.285199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.285324 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.285620 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.285625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.285735 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.285823 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.285939 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260925 05:05:37.286141 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.276323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:47865 (local address 127.14.33.62:46657)
0925 05:05:37.276399 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.276401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.276535 (+ 134us) server_negotiation.cc:407] Connection header received
0925 05:05:37.276988 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.276989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.277006 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.277083 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.277983 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.278809 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.280404 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.280757 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.280846 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.283399 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.283400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.283401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.283408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.284545 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.285026 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.285027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.285028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.285111 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.285406 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.285407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.285407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.285547 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.285620 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.285870 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.285969 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:05:37.288442 16395 heartbeater.cc:347] Connected to a master server at 127.14.33.62:46657
I20260925 05:05:37.288524 16395 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:37.288693 16395 heartbeater.cc:511] Master 127.14.33.62:46657 requested a full tablet report, sending...
I20260925 05:05:37.289255 16191 ts_manager.cc:194] Registered new tserver with Master: b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863)
I20260925 05:05:37.290237 16191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:47865
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
I20260925 05:05:37.294655 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.288539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.14.33.1:38983)
0925 05:05:37.288680 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.288691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.288808 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.289013 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.289014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.289223 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.289326 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.289328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.290177 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.290178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.290824 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.290826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.291202 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.291203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.291275 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.291738 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.291742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.292021 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.293569 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.293570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.293570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.293741 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.293987 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.293988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.293989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.294047 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.294309 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.294310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.294365 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.294452 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.294555 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260925 05:05:37.294689 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.288734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:38983 (local address 127.14.33.62:46657)
0925 05:05:37.288770 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.288772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.288780 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:37.288912 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.288913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.288923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.288970 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.289420 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.290079 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.290903 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.291111 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.291199 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.292107 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.292109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.292109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.292116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.293476 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.293860 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.293861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.293861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.293921 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.294128 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.294129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.294129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.294237 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.294319 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.294478 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.294554 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:37.311517 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46657
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:37.387929 16404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:37.388134 16404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:37.388185 16404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:37.389261 16404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:37.389297 16404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:37.389315 16404 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:05:37.389355 16404 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:05:37.389386 16404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:37.390655 16404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:46657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:37.390883 16404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:37.391084 16404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:37.395501 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.395515 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.395515 16404 server_base.cc:1034] running on GCE node
W20260925 05:05:37.395502 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.395888 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:37.396137 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:37.397306 16404 hybrid_clock.cc:648] HybridClock initialized: now 1790312737397296 us; error 40 us; skew 500 ppm
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:37.399355 16404 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:37.399699 16404 webserver.cc:460] Webserver started at http://127.14.33.2:38823/ using document root <none> and password file <none>
I20260925 05:05:37.399893 16404 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:37.399950 16404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:37.400085 16404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:37.400969 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "7c167bb07ad43731477bfc56e414b53b"
server_key_iv: "af3b8d8d2c24fa23a35cc38138819863"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.401335 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "7c167bb07ad43731477bfc56e414b53b"
server_key_iv: "af3b8d8d2c24fa23a35cc38138819863"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.402527 16404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:37.403201 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.403368 16404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:37.403457 16404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "7c167bb07ad43731477bfc56e414b53b"
server_key_iv: "af3b8d8d2c24fa23a35cc38138819863"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.403538 16404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:37.421345 16404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:37.422900 16404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:37.423051 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:37.423491 16404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:37.423549 16404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.423585 16404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:37.423612 16404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.429647 16404 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:35067
I20260925 05:05:37.429718 16534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:35067 every 8 connection(s)
I20260925 05:05:37.429957 16404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:37.432868 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.423973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:49997 (local address 127.14.33.62:46657)
0925 05:05:37.424022 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.424024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.424231 (+ 207us) server_negotiation.cc:407] Connection header received
0925 05:05:37.424660 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.424662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.424675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.424715 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.425554 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.426365 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.427299 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.427618 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.427699 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.430495 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.430497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.430497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.430503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.431798 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.432135 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.432135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.432136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.432196 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.432386 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.432386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.432387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.432477 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.432522 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.432742 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.432808 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:05:37.433004 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.423906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.14.33.2:49997)
0925 05:05:37.424133 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.424156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.424556 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.424795 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.424798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.425124 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.425415 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.425421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.426475 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.426476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.427185 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.427187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.427731 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.427732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.427870 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.428810 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.428820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.430406 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.431863 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.431868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.431870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.432074 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.432247 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.432248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.432248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.432337 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.432527 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.432530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.432646 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.432724 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.432831 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260925 05:05:37.433533 16535 heartbeater.cc:347] Connected to a master server at 127.14.33.62:46657
I20260925 05:05:37.433606 16535 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:37.433774 16535 heartbeater.cc:511] Master 127.14.33.62:46657 requested a full tablet report, sending...
I20260925 05:05:37.434101 16192 ts_manager.cc:194] Registered new tserver with Master: e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067)
I20260925 05:05:37.434465 16192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:49997
I20260925 05:05:37.435736 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 16404
I20260925 05:05:37.435808 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:37.435984 14468 external_mini_cluster.cc:1428] Setting key 563c519a50fe1d1b6d51d67cce3e9f11
I20260925 05:05:37.439786 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.434541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:47081 (local address 127.14.33.62:46657)
0925 05:05:37.434581 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.434583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.434632 (+ 49us) server_negotiation.cc:407] Connection header received
0925 05:05:37.434694 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.434695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.434703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.434751 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.435048 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.435517 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.436232 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.436405 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.436490 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.437379 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.437380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.437380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.437385 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.438817 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.439159 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.439160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.439160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.439204 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.439377 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.439377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.439377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.439487 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.439530 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.439681 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.439728 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:05:37.439919 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.434457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.14.33.2:47081)
0925 05:05:37.434586 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.434595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.434644 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.434788 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.434789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.434922 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.434973 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.434975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.435606 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.435607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.436159 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.436161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.436490 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.436491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.436559 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.436896 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.436900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.437307 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.438891 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.438892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.438892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.439090 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.439259 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.439260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.439261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.439325 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.439542 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.439543 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.439598 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.439669 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.439777 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:37.458948 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46657
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:37.532755 16544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:37.532922 16544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:37.532943 16544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:37.534029 16544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:37.534070 16544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:37.534089 16544 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 05:05:37.534154 16544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:05:37.534183 16544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:37.535563 16544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:46657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:37.535794 16544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:37.536016 16544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:05:37.540733 16544 server_base.cc:1034] running on GCE node
W20260925 05:05:37.540719 16549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.540719 16552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:37.540736 16550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:37.541169 16544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:37.541406 16544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:37.542562 16544 hybrid_clock.cc:648] HybridClock initialized: now 1790312737542550 us; error 38 us; skew 500 ppm
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:37.544719 16544 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:37.545084 16544 webserver.cc:460] Webserver started at http://127.14.33.3:36723/ using document root <none> and password file <none>
I20260925 05:05:37.545318 16544 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:37.545380 16544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:37.545517 16544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:37.546442 16544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "f429eecbb7901552ea6c90e5923ff01b"
server_key_iv: "a334f6185907418f7eb31e3e856145b7"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.546770 16544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "f429eecbb7901552ea6c90e5923ff01b"
server_key_iv: "a334f6185907418f7eb31e3e856145b7"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.547966 16544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:37.548657 16559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.548812 16544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:37.548897 16544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
format_stamp: "Formatted at 2026-09-25 05:05:37 on dist-test-slave-4dmq"
server_key: "f429eecbb7901552ea6c90e5923ff01b"
server_key_iv: "a334f6185907418f7eb31e3e856145b7"
server_key_version: "encryptionkey@0"
I20260925 05:05:37.548970 16544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:37.559110 16544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:37.561069 16544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:37.561271 16544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:37.561782 16544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:37.561820 16544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.561841 16544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:37.561880 16544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.568420 16544 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:45613
I20260925 05:05:37.568449 16674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:45613 every 8 connection(s)
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:37.568782 16544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:37.571373 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.562227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:41559 (local address 127.14.33.62:46657)
0925 05:05:37.562264 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.562266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.562470 (+ 204us) server_negotiation.cc:407] Connection header received
0925 05:05:37.562981 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.562983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.562997 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.563089 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.563845 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.564662 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.566136 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.566485 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.566578 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.569214 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.569216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.569217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.569225 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.570335 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.570664 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.570664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.570665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.570714 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.570906 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.570906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.570907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.571001 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.571060 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.571262 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.571315 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:05:37.571537 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.562218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.14.33.3:41559)
0925 05:05:37.562379 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.562406 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.562889 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.563160 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.563163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.563502 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.563748 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.563754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.564753 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.564754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.566022 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.566025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.566580 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.566581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.566723 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.567247 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.567257 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.569084 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.570397 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.570402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.570403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.570601 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.570768 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.570768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.570769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.570855 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.571062 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.571066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.571175 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.571249 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.571358 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260925 05:05:37.571985 16675 heartbeater.cc:347] Connected to a master server at 127.14.33.62:46657
I20260925 05:05:37.572057 16675 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:37.572225 16675 heartbeater.cc:511] Master 127.14.33.62:46657 requested a full tablet report, sending...
I20260925 05:05:37.572542 16191 ts_manager.cc:194] Registered new tserver with Master: dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:37.572880 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 16544
I20260925 05:05:37.572989 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:37.573014 16191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:41559
I20260925 05:05:37.573220 14468 external_mini_cluster.cc:1428] Setting key de03c4e19dba3f78c046bacfb815da31
I20260925 05:05:37.574018 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:37.578423 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.572910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.14.33.3:59091)
0925 05:05:37.573064 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.573075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.573184 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.573381 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.573382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.573577 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.573636 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.573638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.574367 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.574368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.574849 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.574851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.575184 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.575185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.575251 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.575632 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.575638 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.576032 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.577557 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.577557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.577558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.577699 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.577904 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.577905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.577905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.577963 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.578210 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.578210 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.578250 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.578296 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.578360 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":180,"threads_started":1}
I20260925 05:05:37.578514 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.572874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:59091 (local address 127.14.33.62:46657)
0925 05:05:37.572932 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.572933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.573160 (+ 227us) server_negotiation.cc:407] Connection header received
0925 05:05:37.573233 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.573234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.573243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.573307 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.573717 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.574277 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.574939 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.575109 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.575182 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.576123 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.576124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.576124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.576129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.577465 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.577777 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.577778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.577778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.577835 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.578023 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.578024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.578025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.578148 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.578212 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.578339 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.578411 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:37.595279 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.588986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.14.33.62:46657)
0925 05:05:37.589027 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.589029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.589037 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:37.589111 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.589111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.589134 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.589203 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.589717 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.590178 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.590869 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.591045 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.591115 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.593182 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.593183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.593183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.593190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.594270 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.594603 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.594604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.594605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.594659 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.594839 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.594840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.594840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.594936 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.595030 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.595117 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.595185 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:05:37.596503 16191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:37.598048 16191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:37.603552 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.599910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45613 (local address 127.0.0.1:45296)
0925 05:05:37.600141 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.600150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.600221 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.600439 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.600441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.600443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:37.600495 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.600497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.601424 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.601425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.602431 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.602433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.603257 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.603258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.603342 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.603356 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.603420 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.603475 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":59,"threads_started":1}
I20260925 05:05:37.603629 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.600241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.14.33.1:32863)
0925 05:05:37.600394 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.600398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.600405 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:05:37.600437 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.600438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.600452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.600650 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:37.600710 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.601733 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.602578 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.603263 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.603364 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.603416 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.603489 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.603525 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260925 05:05:37.603868 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.600011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.14.33.3:45613)
0925 05:05:37.600225 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.600230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.600255 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:05:37.600299 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.600301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.600314 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.600379 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:37.600545 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.601351 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.602651 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.603173 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.603254 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.603318 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.603752 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.603787 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":64,"threads_started":1}
I20260925 05:05:37.603957 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.600057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.14.33.2:35067)
0925 05:05:37.600339 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.600343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.600350 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:05:37.600380 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.600381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.600392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.600457 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:37.600583 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.601738 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.602840 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.603525 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.603585 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.603632 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.603809 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.603843 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":70,"threads_started":1}
I20260925 05:05:37.603574 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.600091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32863 (local address 127.0.0.1:42452)
0925 05:05:37.600240 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.600249 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.600302 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.600534 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.600534 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.600536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:37.600566 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.600567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.601845 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.601846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.602513 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.602515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.603368 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.603368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.603428 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.603432 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.603480 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.603509 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":34,"threads_started":1}
I20260925 05:05:37.603911 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.599906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.0.0.1:49220)
0925 05:05:37.600006 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.600041 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.600091 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.600455 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.600455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.600457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:37.600509 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.600512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.601815 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.601816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.602766 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.602769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.603667 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.603669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.603737 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.603741 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.603794 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.603829 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
I20260925 05:05:37.604553 16605 tablet_service.cc:1467] Processing CreateTablet for tablet fef14973b46c442bbf1c84606c32a376 (DEFAULT_TABLE table=test-table [id=4d2033a2014647b08f85dc185c01e350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:37.604635 16469 tablet_service.cc:1467] Processing CreateTablet for tablet fef14973b46c442bbf1c84606c32a376 (DEFAULT_TABLE table=test-table [id=4d2033a2014647b08f85dc185c01e350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:37.604892 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef14973b46c442bbf1c84606c32a376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.604948 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef14973b46c442bbf1c84606c32a376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.605507 16329 tablet_service.cc:1467] Processing CreateTablet for tablet fef14973b46c442bbf1c84606c32a376 (DEFAULT_TABLE table=test-table [id=4d2033a2014647b08f85dc185c01e350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:37.605808 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef14973b46c442bbf1c84606c32a376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.607419 16695 tablet_bootstrap.cc:492] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Bootstrap starting.
I20260925 05:05:37.608342 16696 tablet_bootstrap.cc:492] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Bootstrap starting.
I20260925 05:05:37.608491 16695 tablet_bootstrap.cc:654] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.608844 16695 log.cc:824] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:37.608935 16696 tablet_bootstrap.cc:654] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.609297 16698 tablet_bootstrap.cc:492] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Bootstrap starting.
I20260925 05:05:37.609311 16696 log.cc:824] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:37.609884 16698 tablet_bootstrap.cc:654] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.609908 16695 tablet_bootstrap.cc:492] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: No bootstrap required, opened a new log
I20260925 05:05:37.609965 16696 tablet_bootstrap.cc:492] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: No bootstrap required, opened a new log
I20260925 05:05:37.609994 16695 ts_tablet_manager.cc:1397] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 05:05:37.610031 16696 ts_tablet_manager.cc:1397] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:05:37.610209 16698 log.cc:824] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:37.610908 16698 tablet_bootstrap.cc:492] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: No bootstrap required, opened a new log
I20260925 05:05:37.610975 16698 ts_tablet_manager.cc:1397] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:37.611654 16695 raft_consensus.cc:348] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.611804 16695 raft_consensus.cc:374] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.611845 16695 raft_consensus.cc:729] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6e5628b6a94f869d4606c638a7f2e8, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.611967 16695 consensus_queue.cc:260] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.612205 16695 ts_tablet_manager.cc:1428] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:37.612082 16696 raft_consensus.cc:348] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.612260 16696 raft_consensus.cc:374] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.612304 16696 raft_consensus.cc:729] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.612336 16675 heartbeater.cc:503] Master 127.14.33.62:46657 was elected leader, sending a full tablet report...
I20260925 05:05:37.612414 16696 consensus_queue.cc:260] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.612658 16696 ts_tablet_manager.cc:1428] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:05:37.612676 16535 heartbeater.cc:503] Master 127.14.33.62:46657 was elected leader, sending a full tablet report...
I20260925 05:05:37.613524 16698 raft_consensus.cc:348] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.613629 16698 raft_consensus.cc:374] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.613646 16698 raft_consensus.cc:729] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.613798 16698 consensus_queue.cc:260] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.614077 16698 ts_tablet_manager.cc:1428] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:37.614111 16395 heartbeater.cc:503] Master 127.14.33.62:46657 was elected leader, sending a full tablet report...
W20260925 05:05:37.680617 16536 tablet.cc:2369] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:37.783470 16396 tablet.cc:2369] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:37.819295 16676 tablet.cc:2369] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:37.836768 16701 raft_consensus.cc:482] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:37.836871 16701 raft_consensus.cc:504] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.837229 16701 leader_election.cc:288] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067), b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863)
I20260925 05:05:37.839798 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.837398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.14.33.3:40241)
0925 05:05:37.837452 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.837464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.837507 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.837633 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.837634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.837635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:37.837705 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.837707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.838326 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.838327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.839022 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.839024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.839546 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.839547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.839626 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.839629 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.839691 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.839728 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:05:37.839798 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.837440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:40241 (local address 127.14.33.2:35067)
0925 05:05:37.837484 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.837486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.837494 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:37.837569 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.837570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.837582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.837630 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:37.837795 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.838244 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.839113 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.839460 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.839538 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.839547 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.839697 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.839731 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:05:37.840008 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.837500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:32817 (local address 127.14.33.1:32863)
0925 05:05:37.837543 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.837545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.837684 (+ 139us) server_negotiation.cc:407] Connection header received
0925 05:05:37.837746 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.837747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.837760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.837830 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:37.837970 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.838421 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.839293 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.839641 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.839730 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.839739 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.839888 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.839928 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:05:37.840026 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.837405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32863 (local address 127.14.33.3:32817)
0925 05:05:37.837611 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.837619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.837691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.837831 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.837832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.837834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:37.837904 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.837905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.838514 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.838515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.839195 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.839196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.839732 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.839733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.839809 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.839812 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.839870 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.839905 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260925 05:05:37.840145 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef14973b46c442bbf1c84606c32a376" candidate_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" is_pre_election: true
I20260925 05:05:37.840286 16487 raft_consensus.cc:2454] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd6e5628b6a94f869d4606c638a7f2e8 in term 0.
I20260925 05:05:37.840360 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef14973b46c442bbf1c84606c32a376" candidate_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24f728225bb4a7e9494d4ecca09fec3" is_pre_election: true
I20260925 05:05:37.840494 16349 raft_consensus.cc:2454] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd6e5628b6a94f869d4606c638a7f2e8 in term 0.
I20260925 05:05:37.840502 16561 leader_election.cc:302] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd6e5628b6a94f869d4606c638a7f2e8, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:37.840615 16701 raft_consensus.cc:2781] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:37.840670 16701 raft_consensus.cc:482] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:37.840694 16701 raft_consensus.cc:3037] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:37.841377 16701 raft_consensus.cc:504] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.841508 16701 leader_election.cc:288] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [CANDIDATE]: Term 1 election: Requested vote from peers e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067), b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863)
I20260925 05:05:37.841634 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef14973b46c442bbf1c84606c32a376" candidate_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c"
I20260925 05:05:37.841723 16487 raft_consensus.cc:3037] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:37.841770 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef14973b46c442bbf1c84606c32a376" candidate_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24f728225bb4a7e9494d4ecca09fec3"
I20260925 05:05:37.841885 16349 raft_consensus.cc:3037] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:37.842471 16487 raft_consensus.cc:2454] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd6e5628b6a94f869d4606c638a7f2e8 in term 1.
I20260925 05:05:37.842626 16561 leader_election.cc:302] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd6e5628b6a94f869d4606c638a7f2e8, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:37.842720 16701 raft_consensus.cc:2781] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:37.842794 16349 raft_consensus.cc:2454] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd6e5628b6a94f869d4606c638a7f2e8 in term 1.
I20260925 05:05:37.842844 16701 raft_consensus.cc:686] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 LEADER]: Becoming Leader. State: Replica: dd6e5628b6a94f869d4606c638a7f2e8, State: Running, Role: LEADER
I20260925 05:05:37.843026 16701 consensus_queue.cc:237] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } }
I20260925 05:05:37.843806 16191 catalog_manager.cc:5495] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:05:37 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:37.903991 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.897772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.14.33.62:46657)
0925 05:05:37.897860 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:37.897863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:37.897967 (+ 104us) server_negotiation.cc:407] Connection header received
0925 05:05:37.898059 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:37.898060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:37.898069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:37.898142 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:37.898583 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.899066 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.899735 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.899923 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.900020 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.901855 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:37.901856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:37.901856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:37.901863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:37.902895 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.903262 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.903263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.903264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.903314 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:37.903489 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:37.903490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:37.903490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:37.903603 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:37.903672 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:37.903811 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 05:05:37.903886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 05:05:37.904053 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:37.897701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46657 (local address 127.0.0.1:42020)
0925 05:05:37.897879 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:37.897896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:37.897998 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:37.898159 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:37.898159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:37.898462 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:37.898501 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.898504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.899147 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.899148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.899647 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:37.899649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:37.900020 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:37.900021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:37.900082 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:37.900532 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:37.900536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:37.901775 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:37.902981 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.902988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.902991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.903187 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.903371 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:37.903372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:37.903372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:37.903435 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:37.903663 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:37.903669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:37.903710 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:05:37.903790 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:05:37.903946 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":70,"thread_start_us":84,"threads_started":1}
I20260925 05:05:37.905906 16191 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42020:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:05:37.906091 16191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:37.907389 16329 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2402d120464996b327d345c3da88f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44703419f7f54003947cdc402b492050]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:05:37.907401 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2402d120464996b327d345c3da88f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44703419f7f54003947cdc402b492050]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:05:37.907559 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2402d120464996b327d345c3da88f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.907567 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2402d120464996b327d345c3da88f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.907670 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2402d120464996b327d345c3da88f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44703419f7f54003947cdc402b492050]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:05:37.907801 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2402d120464996b327d345c3da88f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:37.908574 16696 tablet_bootstrap.cc:492] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: Bootstrap starting.
I20260925 05:05:37.908960 16698 tablet_bootstrap.cc:492] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: Bootstrap starting.
I20260925 05:05:37.909142 16695 tablet_bootstrap.cc:492] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: Bootstrap starting.
I20260925 05:05:37.909163 16696 tablet_bootstrap.cc:654] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.909339 16698 tablet_bootstrap.cc:654] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.909557 16695 tablet_bootstrap.cc:654] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:37.909971 16698 tablet_bootstrap.cc:492] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: No bootstrap required, opened a new log
I20260925 05:05:37.910020 16698 ts_tablet_manager.cc:1397] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:37.910123 16695 tablet_bootstrap.cc:492] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: No bootstrap required, opened a new log
I20260925 05:05:37.910133 16696 tablet_bootstrap.cc:492] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: No bootstrap required, opened a new log
I20260925 05:05:37.910178 16696 ts_tablet_manager.cc:1397] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:05:37.910161 16698 raft_consensus.cc:348] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:37.910174 16695 ts_tablet_manager.cc:1397] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:37.910204 16698 raft_consensus.cc:374] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.910216 16698 raft_consensus.cc:729] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.910267 16698 consensus_queue.cc:260] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:37.910310 16696 raft_consensus.cc:348] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:37.910310 16695 raft_consensus.cc:348] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:37.910356 16698 ts_tablet_manager.cc:1428] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.910367 16696 raft_consensus.cc:374] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.910375 16695 raft_consensus.cc:374] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:37.910394 16695 raft_consensus.cc:729] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6e5628b6a94f869d4606c638a7f2e8, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.910390 16696 raft_consensus.cc:729] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Initialized, Role: FOLLOWER
I20260925 05:05:37.910395 16703 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:37.910432 16696 consensus_queue.cc:260] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:37.910431 16695 consensus_queue.cc:260] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:37.910468 16703 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:37.910503 16696 ts_tablet_manager.cc:1428] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:37.910522 16695 ts_tablet_manager.cc:1428] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:37.910648 16702 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:37.910706 16701 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:37.910742 16702 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:37.910768 16701 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:38.205375 16702 raft_consensus.cc:482] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:38.205459 16702 raft_consensus.cc:504] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.205813 16702 leader_election.cc:288] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:38.209817 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.206035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:57243 (local address 127.14.33.1:32863)
0925 05:05:38.206099 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.206101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.206189 (+ 88us) server_negotiation.cc:407] Connection header received
0925 05:05:38.206226 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.206227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.206239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.206327 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:38.206584 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.207352 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.208875 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.209470 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.209554 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.209564 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.209714 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.209762 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:05:38.209854 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.205962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45613 (local address 127.14.33.2:56271)
0925 05:05:38.206107 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.206119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.206174 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.206391 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.206392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.206396 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:38.206509 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.206511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.207434 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.207435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.208787 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.208790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.209563 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.209564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.209633 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.209637 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.209691 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.209738 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":85,"thread_start_us":57,"threads_started":1}
I20260925 05:05:38.209860 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.205962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32863 (local address 127.14.33.2:57243)
0925 05:05:38.206107 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.206119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.206176 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.206391 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.206392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.206396 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:38.206495 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.206498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.207435 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.207436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.208787 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.208802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.209562 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.209563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.209633 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.209639 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.209692 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.209738 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":99,"thread_start_us":60,"threads_started":1}
I20260925 05:05:38.210067 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.206036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:56271 (local address 127.14.33.3:45613)
0925 05:05:38.206149 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.206151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.206162 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:05:38.206252 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.206253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.206264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.206350 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:38.206583 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.207368 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.208875 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.209481 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.209554 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.209564 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.209935 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.209988 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260925 05:05:38.210150 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2402d120464996b327d345c3da88f7" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24f728225bb4a7e9494d4ecca09fec3" is_pre_election: true
I20260925 05:05:38.210239 16349 raft_consensus.cc:2454] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 0.
I20260925 05:05:38.210305 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2402d120464996b327d345c3da88f7" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" is_pre_election: true
I20260925 05:05:38.210420 16629 raft_consensus.cc:2454] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 0.
I20260925 05:05:38.210454 16423 leader_election.cc:302] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24f728225bb4a7e9494d4ecca09fec3, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:38.210578 16702 raft_consensus.cc:2781] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:38.210616 16702 raft_consensus.cc:482] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:38.210640 16702 raft_consensus.cc:3037] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.211267 16702 raft_consensus.cc:504] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.211408 16702 leader_election.cc:288] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 election: Requested vote from peers b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:38.211552 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2402d120464996b327d345c3da88f7" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24f728225bb4a7e9494d4ecca09fec3"
I20260925 05:05:38.211621 16349 raft_consensus.cc:3037] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.212190 16349 raft_consensus.cc:2454] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 1.
I20260925 05:05:38.212301 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2402d120464996b327d345c3da88f7" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
I20260925 05:05:38.212378 16629 raft_consensus.cc:3037] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.212354 16423 leader_election.cc:302] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24f728225bb4a7e9494d4ecca09fec3, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:38.212447 16702 raft_consensus.cc:2781] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:38.212558 16702 raft_consensus.cc:686] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 LEADER]: Becoming Leader. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Running, Role: LEADER
I20260925 05:05:38.212643 16702 consensus_queue.cc:237] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.213063 16719 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: TxnStatusTablet state changed. Reason: New leader e7e9324a6ccb4a9eb56cd886c7b7380c. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.213177 16629 raft_consensus.cc:2454] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 1.
I20260925 05:05:38.213248 16719 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:05:38.213546 16721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:05:38.213657 16721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:05:38.213665 16191 catalog_manager.cc:5495] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c reported cstate change: term changed from 0 to 1, leader changed from <none> to e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2). New cstate: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:38.233251 16701 consensus_queue.cc:1035] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:38.234987 16701 consensus_queue.cc:1035] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:38.264190 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.261599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.14.33.2:35067)
0925 05:05:38.261658 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.261660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.261669 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:05:38.261746 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.261747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.261764 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.261820 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:38.261951 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.262459 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.263355 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.263608 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.263675 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.263863 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.263977 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.264061 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.264100 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.264119 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:05:38.264209 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.261530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.0.0.1:49234)
0925 05:05:38.261600 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.261614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.261664 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.261824 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.261825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.261827 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:38.261876 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.261880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.262536 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.262537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.263277 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.263279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.263674 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.263675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.263739 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.263754 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.264039 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.264048 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.264095 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.264141 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 05:05:38.265873 16629 raft_consensus.cc:1261] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Refusing update from remote peer e7e9324a6ccb4a9eb56cd886c7b7380c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:38.265883 16349 raft_consensus.cc:1261] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Refusing update from remote peer e7e9324a6ccb4a9eb56cd886c7b7380c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:38.266125 16702 consensus_queue.cc:1035] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:38.266259 16719 consensus_queue.cc:1035] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:38.267107 16701 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: TxnStatusTablet state changed. Reason: New leader e7e9324a6ccb4a9eb56cd886c7b7380c. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.267217 16701 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:38.267349 16728 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: TxnStatusTablet state changed. Reason: New leader e7e9324a6ccb4a9eb56cd886c7b7380c. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.267437 16728 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:38.268263 16719 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.268347 16719 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:05:38.268572 16724 mvcc.cc:205] Tried to move back new op lower bound from 7333120975934758912 to 7333120975719563264. Current Snapshot: MvccSnapshot[applied={T|T < 7333120975934758912}]
I20260925 05:05:38.268648 16701 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.268707 16701 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P dd6e5628b6a94f869d4606c638a7f2e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:38.268987 16727 mvcc.cc:205] Tried to move back new op lower bound from 7333120975934758912 to 7333120975719563264. Current Snapshot: MvccSnapshot[applied={T|T < 7333120975934758912}]
I20260925 05:05:38.269052 16728 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.269145 16728 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P b24f728225bb4a7e9494d4ecca09fec3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:05:38.268275 16732 tablet_replica.cc:439] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } } }
I20260925 05:05:38.269603 16732 tablet_replica.cc:442] T 0a2402d120464996b327d345c3da88f7 P e7e9324a6ccb4a9eb56cd886c7b7380c: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:05:38.272821 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.270173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.14.33.3:45613)
0925 05:05:38.270229 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.270231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.270358 (+ 127us) server_negotiation.cc:407] Connection header received
0925 05:05:38.270429 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.270429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.270442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.270494 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:38.270614 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.271041 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.272095 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.272265 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.272317 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.272492 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.272602 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.272656 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.272736 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.272752 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:05:38.277606 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.275142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.14.33.3:35993)
0925 05:05:38.275275 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.275284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.275346 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.275445 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.275446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.275446 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:38.275484 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.275485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.276274 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.276274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.276812 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.276813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.277247 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.277247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.277302 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.277312 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.277484 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.277485 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.277518 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.277549 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260925 05:05:38.277606 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.275222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:35993 (local address 127.14.33.2:35067)
0925 05:05:38.275255 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.275257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.275355 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:05:38.275377 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.275377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.275385 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.275444 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:38.275516 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.276191 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.276888 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.277179 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.277248 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.277396 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.277426 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.277479 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.277520 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.277539 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:05:38.288364 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.285513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:40721 (local address 127.14.33.3:45613)
0925 05:05:38.285547 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.285548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.285664 (+ 116us) server_negotiation.cc:407] Connection header received
0925 05:05:38.285732 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.285733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.285740 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.285783 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:38.285902 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.286518 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.287562 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.287878 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.287951 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.288138 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.288185 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.288229 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.288285 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.288303 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:05:38.288424 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.285462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45613 (local address 127.14.33.2:40721)
0925 05:05:38.285613 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.285623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.285679 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.285787 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.285788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.285789 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:38.285845 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.285847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.286628 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.286628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.287474 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.287477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.287962 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.287963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.288028 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.288042 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.288231 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.288233 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.288281 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.288328 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260925 05:05:38.300761 16192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:38.300881 16192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:38.301625 16192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Sending DeleteTablet for 3 replicas of tablet fef14973b46c442bbf1c84606c32a376
I20260925 05:05:38.301937 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet fef14973b46c442bbf1c84606c32a376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:38 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42452
I20260925 05:05:38.301994 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet fef14973b46c442bbf1c84606c32a376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:38 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45296
I20260925 05:05:38.302181 16746 tablet_replica.cc:330] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: stopping tablet replica
I20260925 05:05:38.302279 16746 raft_consensus.cc:2229] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:38.302335 16746 raft_consensus.cc:2258] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:38.302646 16469 tablet_service.cc:1514] Processing DeleteTablet for tablet fef14973b46c442bbf1c84606c32a376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:38 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:49220
I20260925 05:05:38.302774 16746 ts_tablet_manager.cc:1905] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:38.302907 16192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:38.303059 16192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:38.304598 16746 ts_tablet_manager.cc:1918] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:38.304666 16329 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 (DEFAULT_TABLE table=test-table [id=84a5d819b7884529b585855547ba9e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:38.304695 16746 log.cc:1196] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/fef14973b46c442bbf1c84606c32a376
I20260925 05:05:38.304780 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:38.304982 16746 ts_tablet_manager.cc:1939] T fef14973b46c442bbf1c84606c32a376 P b24f728225bb4a7e9494d4ecca09fec3: Deleting consensus metadata
I20260925 05:05:38.305351 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 (DEFAULT_TABLE table=test-table [id=84a5d819b7884529b585855547ba9e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:38.305446 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:38.306507 16695 tablet_bootstrap.cc:492] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Bootstrap starting.
I20260925 05:05:38.306591 16179 catalog_manager.cc:4841] TS b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863): tablet fef14973b46c442bbf1c84606c32a376 (table test-table [id=4d2033a2014647b08f85dc185c01e350]) successfully deleted
I20260925 05:05:38.306953 16695 tablet_bootstrap.cc:654] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:38.307045 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 (DEFAULT_TABLE table=test-table [id=84a5d819b7884529b585855547ba9e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:38.307150 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:38.308054 16696 tablet_bootstrap.cc:492] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Bootstrap starting.
I20260925 05:05:38.308442 16696 tablet_bootstrap.cc:654] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:38.309139 16698 tablet_bootstrap.cc:492] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Bootstrap starting.
I20260925 05:05:38.309469 16747 tablet_replica.cc:330] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: stopping tablet replica
I20260925 05:05:38.309518 16698 tablet_bootstrap.cc:654] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:38.309569 16747 raft_consensus.cc:2229] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:38.309671 16747 raft_consensus.cc:2258] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:38.309813 16748 tablet_replica.cc:330] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: stopping tablet replica
I20260925 05:05:38.309928 16748 raft_consensus.cc:2229] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:38.310000 16748 raft_consensus.cc:2258] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:38.310349 16748 ts_tablet_manager.cc:1905] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:38.311126 16747 ts_tablet_manager.cc:1905] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:38.312386 16747 ts_tablet_manager.cc:1918] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:38.312501 16747 log.cc:1196] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/fef14973b46c442bbf1c84606c32a376
I20260925 05:05:38.312976 16747 ts_tablet_manager.cc:1939] T fef14973b46c442bbf1c84606c32a376 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting consensus metadata
I20260925 05:05:38.313400 16178 catalog_manager.cc:4841] TS dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613): tablet fef14973b46c442bbf1c84606c32a376 (table test-table [id=4d2033a2014647b08f85dc185c01e350]) successfully deleted
I20260925 05:05:38.313653 16748 ts_tablet_manager.cc:1918] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:38.313735 16748 log.cc:1196] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/fef14973b46c442bbf1c84606c32a376
I20260925 05:05:38.313963 16748 ts_tablet_manager.cc:1939] T fef14973b46c442bbf1c84606c32a376 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting consensus metadata
I20260925 05:05:38.314354 16177 catalog_manager.cc:4841] TS e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067): tablet fef14973b46c442bbf1c84606c32a376 (table test-table [id=4d2033a2014647b08f85dc185c01e350]) successfully deleted
I20260925 05:05:38.314582 16696 tablet_bootstrap.cc:492] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: No bootstrap required, opened a new log
I20260925 05:05:38.314643 16696 ts_tablet_manager.cc:1397] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 05:05:38.314767 16696 raft_consensus.cc:348] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.314814 16696 raft_consensus.cc:374] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:38.314827 16696 raft_consensus.cc:729] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Initialized, Role: FOLLOWER
I20260925 05:05:38.314880 16696 consensus_queue.cc:260] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.314981 16696 ts_tablet_manager.cc:1428] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.315565 16698 tablet_bootstrap.cc:492] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: No bootstrap required, opened a new log
I20260925 05:05:38.315615 16698 ts_tablet_manager.cc:1397] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 05:05:38.315752 16698 raft_consensus.cc:348] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.315846 16698 raft_consensus.cc:374] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:38.315871 16698 raft_consensus.cc:729] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Initialized, Role: FOLLOWER
I20260925 05:05:38.315912 16698 consensus_queue.cc:260] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.315974 16695 tablet_bootstrap.cc:492] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: No bootstrap required, opened a new log
I20260925 05:05:38.315999 16698 ts_tablet_manager.cc:1428] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.316018 16695 ts_tablet_manager.cc:1397] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 05:05:38.316151 16695 raft_consensus.cc:348] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.316227 16695 raft_consensus.cc:374] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:38.316248 16695 raft_consensus.cc:729] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6e5628b6a94f869d4606c638a7f2e8, State: Initialized, Role: FOLLOWER
I20260925 05:05:38.316288 16695 consensus_queue.cc:260] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.316375 16695 ts_tablet_manager.cc:1428] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:38.379244 16719 raft_consensus.cc:482] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:38.379348 16719 raft_consensus.cc:504] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.379506 16719 leader_election.cc:288] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:38.379665 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c9ec67ba84f8ead1f8b17b1f521d9" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24f728225bb4a7e9494d4ecca09fec3" is_pre_election: true
I20260925 05:05:38.379665 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c9ec67ba84f8ead1f8b17b1f521d9" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" is_pre_election: true
I20260925 05:05:38.379770 16349 raft_consensus.cc:2454] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 0.
I20260925 05:05:38.379770 16629 raft_consensus.cc:2454] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 0.
I20260925 05:05:38.379922 16422 leader_election.cc:302] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd6e5628b6a94f869d4606c638a7f2e8, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:38.380018 16719 raft_consensus.cc:2781] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:38.380060 16719 raft_consensus.cc:482] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:38.380074 16719 raft_consensus.cc:3037] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.380897 16719 raft_consensus.cc:504] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.381032 16719 leader_election.cc:288] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 election: Requested vote from peers b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:38.381217 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c9ec67ba84f8ead1f8b17b1f521d9" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b24f728225bb4a7e9494d4ecca09fec3"
I20260925 05:05:38.381253 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2c9ec67ba84f8ead1f8b17b1f521d9" candidate_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
I20260925 05:05:38.381307 16349 raft_consensus.cc:3037] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.381312 16629 raft_consensus.cc:3037] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:38.382011 16629 raft_consensus.cc:2454] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 1.
I20260925 05:05:38.382011 16349 raft_consensus.cc:2454] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e9324a6ccb4a9eb56cd886c7b7380c in term 1.
I20260925 05:05:38.382189 16422 leader_election.cc:302] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd6e5628b6a94f869d4606c638a7f2e8, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:38.382282 16719 raft_consensus.cc:2781] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:38.382345 16719 raft_consensus.cc:686] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 LEADER]: Becoming Leader. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Running, Role: LEADER
I20260925 05:05:38.382397 16719 consensus_queue.cc:237] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:38.382930 16192 catalog_manager.cc:5495] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c reported cstate change: term changed from 0 to 1, leader changed from <none> to e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2). New cstate: current_term: 1 leader_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:38.406663 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.404475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.14.33.2:35067)
0925 05:05:38.404519 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.404520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.404531 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:05:38.404555 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.404556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.404568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.404638 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:38.404713 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.405348 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.406034 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.406208 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.406275 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.406400 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.406448 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.406501 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.406572 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.406589 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:05:38.410795 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.408615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:55727 (local address 127.14.33.2:35067)
0925 05:05:38.408649 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:38.408651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:38.408688 (+ 37us) server_negotiation.cc:407] Connection header received
0925 05:05:38.408707 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:38.408708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:38.408717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:38.408776 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:38.408847 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.409449 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.410236 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.410399 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.410464 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.410582 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.410603 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.410635 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:38.410700 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:05:38.410719 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:05:38.410848 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:38.408555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.14.33.2:55727)
0925 05:05:38.408606 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:38.408615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:38.408649 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:38.408776 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:38.408777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:38.408778 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:38.408810 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.408811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.409531 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.409532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.410151 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:38.410153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:38.410487 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:38.410488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:38.410525 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:38.410529 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:38.410649 (+ 120us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:38.410650 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:05:38.410686 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 05:05:38.410722 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260925 05:05:38.412648 16629 raft_consensus.cc:1261] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Refusing update from remote peer e7e9324a6ccb4a9eb56cd886c7b7380c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:38.412725 16349 raft_consensus.cc:1261] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Refusing update from remote peer e7e9324a6ccb4a9eb56cd886c7b7380c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:38.412854 16732 consensus_queue.cc:1035] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:38.412974 16732 consensus_queue.cc:1035] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:38.413985 16725 mvcc.cc:205] Tried to move back new op lower bound from 7333120976536752128 to 7333120976414466048. Current Snapshot: MvccSnapshot[applied={T|T < 7333120976536752128}]
W20260925 05:05:38.417241 16725 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:05:39.936302 16191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:39.936421 16191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:39.937260 16191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Sending DeleteTablet for 3 replicas of tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9
I20260925 05:05:39.937515 16469 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:49220
I20260925 05:05:39.937556 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42452
I20260925 05:05:39.937592 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:39 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45296
I20260925 05:05:39.937752 16775 tablet_replica.cc:330] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: stopping tablet replica
I20260925 05:05:39.937809 16775 raft_consensus.cc:2229] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:39.937903 16775 raft_consensus.cc:2258] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:39.938055 16777 tablet_replica.cc:330] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: stopping tablet replica
I20260925 05:05:39.938122 16777 raft_consensus.cc:2229] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:39.938153 16777 raft_consensus.cc:2258] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:39.938159 16775 ts_tablet_manager.cc:1905] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:39.938310 16776 tablet_replica.cc:330] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: stopping tablet replica
I20260925 05:05:39.938472 16777 ts_tablet_manager.cc:1905] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:39.939838 16775 ts_tablet_manager.cc:1918] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:05:39.939947 16775 log.cc:1196] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/7c2c9ec67ba84f8ead1f8b17b1f521d9
I20260925 05:05:39.940204 16777 ts_tablet_manager.cc:1918] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:05:39.940269 16777 log.cc:1196] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/7c2c9ec67ba84f8ead1f8b17b1f521d9
I20260925 05:05:39.940271 16775 ts_tablet_manager.cc:1939] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting consensus metadata
I20260925 05:05:39.940461 16777 ts_tablet_manager.cc:1939] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting consensus metadata
I20260925 05:05:39.940615 16776 raft_consensus.cc:2229] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:39.940670 16776 raft_consensus.cc:2258] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:39.940750 16177 catalog_manager.cc:4841] TS e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067): tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 (table test-table [id=84a5d819b7884529b585855547ba9e5c]) successfully deleted
I20260925 05:05:39.940939 16178 catalog_manager.cc:4841] TS dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613): tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 (table test-table [id=84a5d819b7884529b585855547ba9e5c]) successfully deleted
I20260925 05:05:39.940986 16776 ts_tablet_manager.cc:1905] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:39.942606 16776 ts_tablet_manager.cc:1918] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:05:39.942674 16776 log.cc:1196] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/7c2c9ec67ba84f8ead1f8b17b1f521d9
I20260925 05:05:39.942929 16776 ts_tablet_manager.cc:1939] T 7c2c9ec67ba84f8ead1f8b17b1f521d9 P b24f728225bb4a7e9494d4ecca09fec3: Deleting consensus metadata
I20260925 05:05:39.943292 16179 catalog_manager.cc:4841] TS b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863): tablet 7c2c9ec67ba84f8ead1f8b17b1f521d9 (table test-table [id=84a5d819b7884529b585855547ba9e5c]) successfully deleted
Sep 25 05:05:39 dist-test-slave-4dmq krb5kdc[16147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:05:39 dist-test-slave-4dmq krb5kdc[16147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:39.960779 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:39.954149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.14.33.62:46657)
0925 05:05:39.954281 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:39.954283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:39.954292 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:05:39.954358 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:39.954358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:39.954368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:39.954428 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:39.954926 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.955400 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.956095 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:39.956289 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:39.956370 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:39.958433 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:39.958435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:39.958435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:39.958441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:39.959934 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.960224 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.960225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.960225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.960276 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:39.960419 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:39.960419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:39.960419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:39.960508 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:39.960573 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:39.960654 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 05:05:39.960719 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":88,"threads_started":1}
I20260925 05:05:39.962258 16191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:39.962417 16191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:39.963570 16329 tablet_service.cc:1467] Processing CreateTablet for tablet bd68ffac4dfe4943ace1bec717d6a404 (DEFAULT_TABLE table=test-table [id=0b04fb8d25cc49429c1fdb7a4bb2fd91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:39.963591 16469 tablet_service.cc:1467] Processing CreateTablet for tablet bd68ffac4dfe4943ace1bec717d6a404 (DEFAULT_TABLE table=test-table [id=0b04fb8d25cc49429c1fdb7a4bb2fd91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:39.963687 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd68ffac4dfe4943ace1bec717d6a404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:39.963685 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd68ffac4dfe4943ace1bec717d6a404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:39.965000 16786 tablet_bootstrap.cc:492] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Bootstrap starting.
I20260925 05:05:39.964998 16605 tablet_service.cc:1467] Processing CreateTablet for tablet bd68ffac4dfe4943ace1bec717d6a404 (DEFAULT_TABLE table=test-table [id=0b04fb8d25cc49429c1fdb7a4bb2fd91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:39.965060 16787 tablet_bootstrap.cc:492] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Bootstrap starting.
I20260925 05:05:39.965102 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd68ffac4dfe4943ace1bec717d6a404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:39.965440 16787 tablet_bootstrap.cc:654] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:39.965437 16786 tablet_bootstrap.cc:654] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:39.966002 16786 tablet_bootstrap.cc:492] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: No bootstrap required, opened a new log
I20260925 05:05:39.966058 16786 ts_tablet_manager.cc:1397] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:39.966180 16787 tablet_bootstrap.cc:492] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: No bootstrap required, opened a new log
I20260925 05:05:39.966163 16786 raft_consensus.cc:348] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:39.966214 16787 ts_tablet_manager.cc:1397] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:39.966228 16786 raft_consensus.cc:374] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:39.966248 16786 raft_consensus.cc:729] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Initialized, Role: FOLLOWER
I20260925 05:05:39.966296 16786 consensus_queue.cc:260] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:39.966404 16786 ts_tablet_manager.cc:1428] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:39.966430 16790 tablet_bootstrap.cc:492] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Bootstrap starting.
I20260925 05:05:39.966738 16787 raft_consensus.cc:348] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:39.966845 16790 tablet_bootstrap.cc:654] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:39.966830 16787 raft_consensus.cc:374] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:39.966991 16787 raft_consensus.cc:729] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Initialized, Role: FOLLOWER
I20260925 05:05:39.967056 16787 consensus_queue.cc:260] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:39.967227 16787 ts_tablet_manager.cc:1428] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:39.967675 16790 tablet_bootstrap.cc:492] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: No bootstrap required, opened a new log
I20260925 05:05:39.967726 16790 ts_tablet_manager.cc:1397] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:39.967844 16790 raft_consensus.cc:348] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:39.967943 16790 raft_consensus.cc:374] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:39.967965 16790 raft_consensus.cc:729] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6e5628b6a94f869d4606c638a7f2e8, State: Initialized, Role: FOLLOWER
I20260925 05:05:39.968014 16790 consensus_queue.cc:260] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:39.968165 16790 ts_tablet_manager.cc:1428] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:40.113648 16792 raft_consensus.cc:482] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:40.113745 16792 raft_consensus.cc:504] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.114130 16792 leader_election.cc:288] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:40.117242 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.114421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.14.33.1:39313)
0925 05:05:40.114623 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:40.114634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:40.114692 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:40.114860 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:40.114861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:40.114863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:40.114939 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.114941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.115568 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.115570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.116387 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.116389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.116947 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.116948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.117023 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.117027 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:05:40.117094 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:05:40.117151 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":136,"threads_started":1}
I20260925 05:05:40.117333 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.114509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:37769 (local address 127.14.33.3:45613)
0925 05:05:40.114663 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.114665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.114673 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:40.114700 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.114701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.114711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.114812 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:40.114972 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.115455 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.116551 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.116929 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.117011 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.117022 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.117224 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.117269 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260925 05:05:40.117342 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.114365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45613 (local address 127.14.33.1:37769)
0925 05:05:40.114563 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:40.114577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:40.114634 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:40.114823 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:40.114824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:40.114827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:40.114900 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.114901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.115684 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.115685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.116455 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.116457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.117015 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.117016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.117092 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.117095 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:40.117212 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 05:05:40.117251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260925 05:05:40.117233 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.114548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:39313 (local address 127.14.33.2:35067)
0925 05:05:40.114685 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.114687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.114696 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:05:40.114759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.114759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.114775 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.114855 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:40.115015 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.115482 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.116485 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.116862 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.116945 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.116956 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.117102 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.117165 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260925 05:05:40.117559 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd68ffac4dfe4943ace1bec717d6a404" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" is_pre_election: true
I20260925 05:05:40.117663 16629 raft_consensus.cc:2454] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 0.
I20260925 05:05:40.117692 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd68ffac4dfe4943ace1bec717d6a404" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" is_pre_election: true
I20260925 05:05:40.117784 16487 raft_consensus.cc:2454] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 0.
I20260925 05:05:40.117851 16282 leader_election.cc:302] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24f728225bb4a7e9494d4ecca09fec3, dd6e5628b6a94f869d4606c638a7f2e8; no voters:
I20260925 05:05:40.118013 16792 raft_consensus.cc:2781] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:40.118043 16792 raft_consensus.cc:482] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:40.118057 16792 raft_consensus.cc:3037] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.118697 16792 raft_consensus.cc:504] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.118834 16792 leader_election.cc:288] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 election: Requested vote from peers e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:40.118999 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd68ffac4dfe4943ace1bec717d6a404" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c"
I20260925 05:05:40.119046 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd68ffac4dfe4943ace1bec717d6a404" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
I20260925 05:05:40.119103 16629 raft_consensus.cc:3037] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.119107 16487 raft_consensus.cc:3037] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.119720 16487 raft_consensus.cc:2454] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 1.
I20260925 05:05:40.119737 16629 raft_consensus.cc:2454] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 1.
I20260925 05:05:40.119920 16281 leader_election.cc:302] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24f728225bb4a7e9494d4ecca09fec3, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:40.120044 16792 raft_consensus.cc:2781] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:40.120206 16792 raft_consensus.cc:686] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 LEADER]: Becoming Leader. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Running, Role: LEADER
I20260925 05:05:40.120289 16792 consensus_queue.cc:237] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.121032 16192 catalog_manager.cc:5495] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1). New cstate: current_term: 1 leader_uuid: "b24f728225bb4a7e9494d4ecca09fec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:40.150414 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.147689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.14.33.1:32863)
0925 05:05:40.147855 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.147856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.147862 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:05:40.147895 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.147896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.147914 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.148001 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:40.148158 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.148665 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.149556 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.149782 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.149854 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.150034 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:40.150150 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:40.150233 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.150272 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.150292 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260925 05:05:40.155112 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.152879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:43373 (local address 127.14.33.2:35067)
0925 05:05:40.152919 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.152921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.152950 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:05:40.153014 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.153014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.153023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.153064 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:40.153285 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.153790 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.154541 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.154755 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.154817 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.154943 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:40.154967 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:40.155002 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.155040 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.155057 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:05:40.155155 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.152748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.14.33.1:43373)
0925 05:05:40.152896 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:40.152906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:40.152964 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:40.153075 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:40.153075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:40.153076 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:40.153150 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.153151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.153856 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.153857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.154472 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.154473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.154817 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.154817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.154863 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.154876 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:40.155003 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:40.155004 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:05:40.155032 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 05:05:40.155066 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260925 05:05:40.157258 16629 raft_consensus.cc:1261] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Refusing update from remote peer b24f728225bb4a7e9494d4ecca09fec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:40.157258 16487 raft_consensus.cc:1261] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Refusing update from remote peer b24f728225bb4a7e9494d4ecca09fec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:40.157516 16798 consensus_queue.cc:1035] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:40.157652 16792 consensus_queue.cc:1035] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:40.164703 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.162261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:48607 (local address 127.14.33.1:32863)
0925 05:05:40.162327 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:40.162329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:40.162376 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:05:40.162502 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:40.162503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:40.162514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:40.162563 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:40.162665 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.163180 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.163956 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.164244 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.164317 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.164460 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:40.164492 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:40.164544 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:40.164614 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:05:40.164636 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:05:40.164710 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:40.162181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32863 (local address 127.14.33.2:48607)
0925 05:05:40.162323 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:40.162335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:40.162392 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:40.162564 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:40.162565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:40.162566 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:05:40.162609 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.162611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.163265 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.163266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.163876 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:40.163879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:40.164316 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:40.164317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:40.164379 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:40.164384 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:05:40.164544 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:05:40.164545 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:05:40.164595 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:05:40.164643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":85,"threads_started":1}
I20260925 05:05:40.170629 16192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:40.170728 16192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:40.171360 16192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Sending DeleteTablet for 3 replicas of tablet bd68ffac4dfe4943ace1bec717d6a404
I20260925 05:05:40.171626 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet bd68ffac4dfe4943ace1bec717d6a404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:40 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42452
I20260925 05:05:40.171633 16469 tablet_service.cc:1514] Processing DeleteTablet for tablet bd68ffac4dfe4943ace1bec717d6a404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:40 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:49220
I20260925 05:05:40.171737 16776 tablet_replica.cc:330] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: stopping tablet replica
I20260925 05:05:40.171741 16775 tablet_replica.cc:330] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: stopping tablet replica
I20260925 05:05:40.171790 16775 raft_consensus.cc:2229] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:40.171792 16776 raft_consensus.cc:2229] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:40.171829 16775 raft_consensus.cc:2258] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:40.171872 16776 raft_consensus.cc:2258] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:40.172021 16192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:40.172174 16192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:40.172212 16775 ts_tablet_manager.cc:1905] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:40.173310 16329 tablet_service.cc:1467] Processing CreateTablet for tablet 13061617219d46d2bc1bf1aad380986a (DEFAULT_TABLE table=test-table [id=1dde9439f64b432aa2c69a6e8b019156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:40.173478 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13061617219d46d2bc1bf1aad380986a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:40.173834 16775 ts_tablet_manager.cc:1918] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:40.173894 16775 log.cc:1196] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/bd68ffac4dfe4943ace1bec717d6a404
I20260925 05:05:40.174105 16775 ts_tablet_manager.cc:1939] T bd68ffac4dfe4943ace1bec717d6a404 P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting consensus metadata
I20260925 05:05:40.174454 16177 catalog_manager.cc:4841] TS e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067): tablet bd68ffac4dfe4943ace1bec717d6a404 (table test-table [id=0b04fb8d25cc49429c1fdb7a4bb2fd91]) successfully deleted
I20260925 05:05:40.174559 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 13061617219d46d2bc1bf1aad380986a (DEFAULT_TABLE table=test-table [id=1dde9439f64b432aa2c69a6e8b019156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:40.174654 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13061617219d46d2bc1bf1aad380986a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:40.174842 16787 tablet_bootstrap.cc:492] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3: Bootstrap starting.
I20260925 05:05:40.175263 16787 tablet_bootstrap.cc:654] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:40.175792 16786 tablet_bootstrap.cc:492] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: Bootstrap starting.
I20260925 05:05:40.175840 16787 tablet_bootstrap.cc:492] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3: No bootstrap required, opened a new log
I20260925 05:05:40.175875 16787 ts_tablet_manager.cc:1397] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:40.176002 16776 ts_tablet_manager.cc:1905] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:40.176031 16787 raft_consensus.cc:348] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.176105 16787 raft_consensus.cc:374] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:40.176110 16786 tablet_bootstrap.cc:654] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:40.176129 16787 raft_consensus.cc:729] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Initialized, Role: FOLLOWER
I20260925 05:05:40.176177 16787 consensus_queue.cc:260] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.176293 16787 ts_tablet_manager.cc:1428] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:40.177342 16776 ts_tablet_manager.cc:1918] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:40.177400 16776 log.cc:1196] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/bd68ffac4dfe4943ace1bec717d6a404
I20260925 05:05:40.177388 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet bd68ffac4dfe4943ace1bec717d6a404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:40 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45296
I20260925 05:05:40.177471 16603 tablet_service.cc:1467] Processing CreateTablet for tablet 13061617219d46d2bc1bf1aad380986a (DEFAULT_TABLE table=test-table [id=1dde9439f64b432aa2c69a6e8b019156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:40.177591 16603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13061617219d46d2bc1bf1aad380986a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:40.177613 16776 ts_tablet_manager.cc:1939] T bd68ffac4dfe4943ace1bec717d6a404 P b24f728225bb4a7e9494d4ecca09fec3: Deleting consensus metadata
I20260925 05:05:40.177794 16777 tablet_replica.cc:330] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: stopping tablet replica
I20260925 05:05:40.177853 16777 raft_consensus.cc:2229] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:40.177886 16777 raft_consensus.cc:2258] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:40.177896 16786 tablet_bootstrap.cc:492] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: No bootstrap required, opened a new log
I20260925 05:05:40.177939 16786 ts_tablet_manager.cc:1397] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:40.178040 16786 raft_consensus.cc:348] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.178097 16786 raft_consensus.cc:374] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:40.178120 16786 raft_consensus.cc:729] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9324a6ccb4a9eb56cd886c7b7380c, State: Initialized, Role: FOLLOWER
I20260925 05:05:40.178164 16786 consensus_queue.cc:260] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.178243 16786 ts_tablet_manager.cc:1428] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:40.178373 16777 ts_tablet_manager.cc:1905] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:40.178503 16179 catalog_manager.cc:4841] TS b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863): tablet bd68ffac4dfe4943ace1bec717d6a404 (table test-table [id=0b04fb8d25cc49429c1fdb7a4bb2fd91]) successfully deleted
I20260925 05:05:40.179061 16790 tablet_bootstrap.cc:492] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Bootstrap starting.
I20260925 05:05:40.179386 16790 tablet_bootstrap.cc:654] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:40.179677 16777 ts_tablet_manager.cc:1918] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:05:40.179749 16777 log.cc:1196] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/bd68ffac4dfe4943ace1bec717d6a404
I20260925 05:05:40.179936 16790 tablet_bootstrap.cc:492] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: No bootstrap required, opened a new log
I20260925 05:05:40.180019 16790 ts_tablet_manager.cc:1397] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:40.180121 16790 raft_consensus.cc:348] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.180202 16790 raft_consensus.cc:374] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:40.180229 16790 raft_consensus.cc:729] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6e5628b6a94f869d4606c638a7f2e8, State: Initialized, Role: FOLLOWER
I20260925 05:05:40.180279 16790 consensus_queue.cc:260] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.180371 16790 ts_tablet_manager.cc:1428] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:40.179965 16777 ts_tablet_manager.cc:1939] T bd68ffac4dfe4943ace1bec717d6a404 P dd6e5628b6a94f869d4606c638a7f2e8: Deleting consensus metadata
I20260925 05:05:40.180689 16178 catalog_manager.cc:4841] TS dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613): tablet bd68ffac4dfe4943ace1bec717d6a404 (table test-table [id=0b04fb8d25cc49429c1fdb7a4bb2fd91]) successfully deleted
I20260925 05:05:40.270870 16798 raft_consensus.cc:482] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:40.270975 16798 raft_consensus.cc:504] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.271211 16798 leader_election.cc:288] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:40.271412 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13061617219d46d2bc1bf1aad380986a" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" is_pre_election: true
I20260925 05:05:40.271524 16487 raft_consensus.cc:2454] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 0.
I20260925 05:05:40.271425 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13061617219d46d2bc1bf1aad380986a" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" is_pre_election: true
I20260925 05:05:40.271588 16629 raft_consensus.cc:2454] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 0.
I20260925 05:05:40.271749 16281 leader_election.cc:302] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24f728225bb4a7e9494d4ecca09fec3, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:40.271853 16798 raft_consensus.cc:2781] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:40.271934 16798 raft_consensus.cc:482] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:40.271968 16798 raft_consensus.cc:3037] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.272915 16798 raft_consensus.cc:504] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.273054 16798 leader_election.cc:288] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 election: Requested vote from peers e7e9324a6ccb4a9eb56cd886c7b7380c (127.14.33.2:35067), dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613)
I20260925 05:05:40.273207 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13061617219d46d2bc1bf1aad380986a" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c"
I20260925 05:05:40.273247 16629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13061617219d46d2bc1bf1aad380986a" candidate_uuid: "b24f728225bb4a7e9494d4ecca09fec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd6e5628b6a94f869d4606c638a7f2e8"
I20260925 05:05:40.273334 16629 raft_consensus.cc:3037] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.273334 16487 raft_consensus.cc:3037] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:40.273936 16487 raft_consensus.cc:2454] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 1.
I20260925 05:05:40.274085 16629 raft_consensus.cc:2454] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24f728225bb4a7e9494d4ecca09fec3 in term 1.
I20260925 05:05:40.274121 16281 leader_election.cc:302] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24f728225bb4a7e9494d4ecca09fec3, e7e9324a6ccb4a9eb56cd886c7b7380c; no voters:
I20260925 05:05:40.274227 16798 raft_consensus.cc:2781] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:40.274281 16798 raft_consensus.cc:686] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [term 1 LEADER]: Becoming Leader. State: Replica: b24f728225bb4a7e9494d4ecca09fec3, State: Running, Role: LEADER
I20260925 05:05:40.274326 16798 consensus_queue.cc:237] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } }
I20260925 05:05:40.274993 16192 catalog_manager.cc:5495] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1). New cstate: current_term: 1 leader_uuid: "b24f728225bb4a7e9494d4ecca09fec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b24f728225bb4a7e9494d4ecca09fec3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 } health_report { overall_health: UNKNOWN } } }
I20260925 05:05:40.294613 16487 raft_consensus.cc:1261] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Refusing update from remote peer b24f728225bb4a7e9494d4ecca09fec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:40.294632 16629 raft_consensus.cc:1261] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Refusing update from remote peer b24f728225bb4a7e9494d4ecca09fec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:40.294849 16809 consensus_queue.cc:1035] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd6e5628b6a94f869d4606c638a7f2e8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:40.294967 16798 consensus_queue.cc:1035] T 13061617219d46d2bc1bf1aad380986a P b24f728225bb4a7e9494d4ecca09fec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e9324a6ccb4a9eb56cd886c7b7380c" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:05:40.299095 16759 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 05:05:41.816999 16191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:41.817111 16191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:41.818045 16191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b572b873be24c4b8cd8bac568a80fb8: Sending DeleteTablet for 3 replicas of tablet 13061617219d46d2bc1bf1aad380986a
I20260925 05:05:41.818346 16469 tablet_service.cc:1514] Processing DeleteTablet for tablet 13061617219d46d2bc1bf1aad380986a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:41 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:49220
I20260925 05:05:41.818367 16603 tablet_service.cc:1514] Processing DeleteTablet for tablet 13061617219d46d2bc1bf1aad380986a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:41 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45296
I20260925 05:05:41.818424 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet 13061617219d46d2bc1bf1aad380986a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:41 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42452
I20260925 05:05:41.818564 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 16264
I20260925 05:05:41.825251 16838 tablet_replica.cc:330] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: stopping tablet replica
I20260925 05:05:41.825232 16839 tablet_replica.cc:330] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: stopping tablet replica
I20260925 05:05:41.825348 16838 raft_consensus.cc:2229] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:41.825387 16839 raft_consensus.cc:2229] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:41.825402 16838 raft_consensus.cc:2258] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:41.825445 16839 raft_consensus.cc:2258] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:41.825657 16838 ts_tablet_manager.cc:1905] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:41.825707 16839 ts_tablet_manager.cc:1905] T 13061617219d46d2bc1bf1aad380986a P e7e9324a6ccb4a9eb56cd886c7b7380c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 05:05:41.826231 16179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:32863 (error 108)
I20260925 05:05:41.826459 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 16404
I20260925 05:05:41.827132 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:41.826593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32863 (local address 127.0.0.1:42464)
0925 05:05:41.827043 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:41.827093 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":51,"threads_started":1}
W20260925 05:05:41.827260 16179 catalog_manager.cc:4584] TS b24f728225bb4a7e9494d4ecca09fec3 (127.14.33.1:32863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13061617219d46d2bc1bf1aad380986a: Network error: Client connection negotiation failed: client connection to 127.14.33.1:32863: connect: Connection refused (error 111)
I20260925 05:05:41.828465 16838 ts_tablet_manager.cc:1918] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 05:05:41.828531 16838 log.cc:1196] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/13061617219d46d2bc1bf1aad380986a
I20260925 05:05:41.828748 16838 ts_tablet_manager.cc:1939] T 13061617219d46d2bc1bf1aad380986a P dd6e5628b6a94f869d4606c638a7f2e8: Deleting consensus metadata
I20260925 05:05:41.829108 16178 catalog_manager.cc:4841] TS dd6e5628b6a94f869d4606c638a7f2e8 (127.14.33.3:45613): tablet 13061617219d46d2bc1bf1aad380986a (table test-table [id=1dde9439f64b432aa2c69a6e8b019156]) successfully deleted
I20260925 05:05:41.832404 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:41.832276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35067 (local address 127.0.0.1:49252)
0925 05:05:41.832349 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:41.832379 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.2:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:05:41.832463 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 16544
I20260925 05:05:41.837889 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 16160
2026-09-25T05:05:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6000 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:41 dist-test-slave-4dmq krb5kdc[16845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:41 dist-test-slave-4dmq krb5kdc[16845](info): set up 2 sockets
Sep 25 05:05:41 dist-test-slave-4dmq krb5kdc[16845](info): commencing operation
krb5kdc: starting...
W20260925 05:05:42.928999 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 05:05:43.191958 14468 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.263s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 05:05:43.507366 14468 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.315s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 05:05:43.774523 14468 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.267s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/test-user.keytab.
W20260925 05:05:44.687584 14468 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.913s user 0.001s sys 0.001s
Sep 25 05:05:44 dist-test-slave-4dmq krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
W20260925 05:05:45.953997 14468 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.33.62: real 1.245s user 0.000s sys 0.003s
I20260925 05:05:45.954463 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38981
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:46.035975 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:46.036162 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:46.036190 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:46.037344 16858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:46.037385 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:46.037400 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:46.037413 16858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:46.037425 16858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:46.038821 16858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:46.038998 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:46.039160 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:46.043952 16863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.043965 16864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.044124 16858 server_base.cc:1034] running on GCE node
W20260925 05:05:46.043965 16866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.044375 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:46.044654 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:46.045800 16858 hybrid_clock.cc:648] HybridClock initialized: now 1790312746045787 us; error 29 us; skew 500 ppm
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:46.048259 16858 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:46.048859 16858 webserver.cc:460] Webserver started at http://127.14.33.62:33959/ using document root <none> and password file <none>
I20260925 05:05:46.049160 16858 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:46.049237 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:46.049376 16858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:46.050473 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "0507783ae1094f928900accef3273842"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "2124e54c964aa99ef3543f128942cb2f"
server_key_iv: "cd7fe5281063197d1164a7d9e648ed5e"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.050837 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "0507783ae1094f928900accef3273842"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "2124e54c964aa99ef3543f128942cb2f"
server_key_iv: "cd7fe5281063197d1164a7d9e648ed5e"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.052229 16858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:46.052994 16873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.053182 16858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:46.053272 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "0507783ae1094f928900accef3273842"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "2124e54c964aa99ef3543f128942cb2f"
server_key_iv: "cd7fe5281063197d1164a7d9e648ed5e"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.053325 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:46.082793 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:46.084888 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:46.085041 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:46.088932 16858 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38981
I20260925 05:05:46.089015 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38981 every 8 connection(s)
I20260925 05:05:46.089376 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:46.090049 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 16858
I20260925 05:05:46.090122 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:46.090312 14468 external_mini_cluster.cc:1428] Setting key 0b0ecf66bc6083b4d97e1538a368e105
I20260925 05:05:46.090394 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:46.094246 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842: Bootstrap starting.
I20260925 05:05:46.094902 16926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:46.095232 16926 log.cc:824] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:46.096084 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842: No bootstrap required, opened a new log
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:46.098639 16926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0507783ae1094f928900accef3273842" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38981 } }
I20260925 05:05:46.098837 16926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:46.098888 16926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0507783ae1094f928900accef3273842, State: Initialized, Role: FOLLOWER
I20260925 05:05:46.099025 16926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0507783ae1094f928900accef3273842" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38981 } }
I20260925 05:05:46.099125 16926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:46.099159 16926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:46.099208 16926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:46.099946 16926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0507783ae1094f928900accef3273842" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38981 } }
I20260925 05:05:46.100059 16926 leader_election.cc:302] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0507783ae1094f928900accef3273842; no voters:
I20260925 05:05:46.100247 16926 leader_election.cc:288] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:46.100282 16931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:46.100401 16931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [term 1 LEADER]: Becoming Leader. State: Replica: 0507783ae1094f928900accef3273842, State: Running, Role: LEADER
I20260925 05:05:46.100512 16926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:46.100530 16931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0507783ae1094f928900accef3273842" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38981 } }
I20260925 05:05:46.100884 16931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0507783ae1094f928900accef3273842. Latest consensus state: current_term: 1 leader_uuid: "0507783ae1094f928900accef3273842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0507783ae1094f928900accef3273842" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38981 } } }
I20260925 05:05:46.100910 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0507783ae1094f928900accef3273842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0507783ae1094f928900accef3273842" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38981 } } }
I20260925 05:05:46.101184 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:46.101063 16931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:46.101845 16940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:46.101917 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.090841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.14.33.62:38981)
0925 05:05:46.091011 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:46.091020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:46.091052 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:05:46.091572 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:46.091579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:46.091628 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:46.091741 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:46.092533 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.093533 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.094613 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.095069 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.095172 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.097888 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:46.097894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:46.097894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:46.097907 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:46.099958 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.100415 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.100418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.100418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.100521 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.100712 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.100713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.100714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.101050 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:46.101269 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:46.101571 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 05:05:46.101706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260925 05:05:46.102522 16940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:46.104012 16940 catalog_manager.cc:1347] Generated new cluster ID: 712b3b71cee24ed18b72ce1147a3602d
I20260925 05:05:46.104059 16940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:46.118084 16940 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:46.118809 16940 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:46.123299 16940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0507783ae1094f928900accef3273842: Generated new TSK 0
I20260925 05:05:46.123479 16940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:46.162020 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38981
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:46.258318 16952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:46.258607 16952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:46.258651 16952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:46.260816 16952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:46.260880 16952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:46.260938 16952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:46.263509 16952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:46.263757 16952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:46.264046 16952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:46.269381 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.269395 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.269629 16952 server_base.cc:1034] running on GCE node
W20260925 05:05:46.269384 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.269951 16952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:46.270270 16952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:46.271463 16952 hybrid_clock.cc:648] HybridClock initialized: now 1790312746271441 us; error 35 us; skew 500 ppm
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:46.274785 16952 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:46.275327 16952 webserver.cc:460] Webserver started at http://127.14.33.1:37423/ using document root <none> and password file <none>
I20260925 05:05:46.275607 16952 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:46.275663 16952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:46.275794 16952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:46.276852 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "bfa9463d522d46028625812591d143bc"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "2f5d8f35f1bf8dec3162324aee5ac648"
server_key_iv: "307299261d052222f8c8bc24a047d4b5"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.277287 16952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "bfa9463d522d46028625812591d143bc"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "2f5d8f35f1bf8dec3162324aee5ac648"
server_key_iv: "307299261d052222f8c8bc24a047d4b5"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.278631 16952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:46.279392 16967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.279568 16952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:46.279657 16952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "bfa9463d522d46028625812591d143bc"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "2f5d8f35f1bf8dec3162324aee5ac648"
server_key_iv: "307299261d052222f8c8bc24a047d4b5"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.279743 16952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:46.290616 16952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:46.292881 16952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:46.293040 16952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:46.293314 16952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:46.293623 16952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:46.293677 16952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.293720 16952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:46.293744 16952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.299508 16952 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:46505
I20260925 05:05:46.299587 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:46505 every 8 connection(s)
I20260925 05:05:46.299881 16952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:46.308506 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.300566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49255 (local address 127.14.33.62:38981)
0925 05:05:46.300621 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:46.300623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:46.300812 (+ 189us) server_negotiation.cc:407] Connection header received
0925 05:05:46.301440 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:46.301441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:46.301452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:46.301516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:46.302478 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.302984 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.303923 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.304122 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.304184 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.306214 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:46.306215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:46.306215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:46.306222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:46.307399 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.307772 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.307772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.307773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.307821 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.308025 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.308025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.308026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.308117 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:46.308165 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:46.308383 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 05:05:46.308443 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:05:46.308698 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.300521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38981 (local address 127.14.33.1:49255)
0925 05:05:46.300677 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:46.300718 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:46.301267 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:46.301594 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:46.301597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:46.302207 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:46.302396 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:46.302401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.303065 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.303065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:46.303840 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:46.303842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.304199 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.304200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:46.304292 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.304725 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:46.304732 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:46.306122 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:46.307483 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:46.307488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:46.307490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:46.307696 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:46.307879 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:46.307880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:46.307880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:46.307972 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:46.308170 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:46.308174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:46.308279 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 05:05:46.308358 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:05:46.308467 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260925 05:05:46.309067 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 16952
I20260925 05:05:46.309190 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:46.309312 17081 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38981
I20260925 05:05:46.309399 17081 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:46.309414 14468 external_mini_cluster.cc:1428] Setting key 0577a51fdb95a7c61b481860c470ec62
I20260925 05:05:46.309615 17081 heartbeater.cc:511] Master 127.14.33.62:38981 requested a full tablet report, sending...
I20260925 05:05:46.310405 16890 ts_manager.cc:194] Registered new tserver with Master: bfa9463d522d46028625812591d143bc (127.14.33.1:46505)
I20260925 05:05:46.311299 16890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:49255
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:46.331759 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38981
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:46.411077 17086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:46.411288 17086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:46.411325 17086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:46.412431 17086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:46.412493 17086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:46.412536 17086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:46.413935 17086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:46.414182 17086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:46.414387 17086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:46.419093 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.419075 17094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.419226 17086 server_base.cc:1034] running on GCE node
W20260925 05:05:46.419302 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.419528 17086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:46.419797 17086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:46.420957 17086 hybrid_clock.cc:648] HybridClock initialized: now 1790312746420937 us; error 32 us; skew 500 ppm
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:46.423323 17086 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:46.423714 17086 webserver.cc:460] Webserver started at http://127.14.33.2:39909/ using document root <none> and password file <none>
I20260925 05:05:46.423938 17086 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:46.424000 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:46.424119 17086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:46.425181 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "cef1d492236843559ee921d14a305332"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "5aec207308b2d624dca883bb7a62dd6d"
server_key_iv: "781cd76f68f055aa08c100c918269de5"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.425549 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "cef1d492236843559ee921d14a305332"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "5aec207308b2d624dca883bb7a62dd6d"
server_key_iv: "781cd76f68f055aa08c100c918269de5"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.426793 17086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:46.427495 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.427673 17086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:46.427762 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "cef1d492236843559ee921d14a305332"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "5aec207308b2d624dca883bb7a62dd6d"
server_key_iv: "781cd76f68f055aa08c100c918269de5"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.427846 17086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:46.447079 17086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:46.448946 17086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:46.449179 17086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:46.449407 17086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:46.449719 17086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:46.449776 17086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.449818 17086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:46.449849 17086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.456102 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:40881 every 8 connection(s)
I20260925 05:05:46.456108 17086 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:40881
I20260925 05:05:46.456468 17086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:46.464985 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.457174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:38991 (local address 127.14.33.62:38981)
0925 05:05:46.457232 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:46.457235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:46.457500 (+ 265us) server_negotiation.cc:407] Connection header received
0925 05:05:46.457979 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:46.457980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:46.457991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:46.458054 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:46.458980 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.459461 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.460381 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.460567 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.460624 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.462619 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:46.462621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:46.462621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:46.462627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:46.463858 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.464209 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.464210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.464210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.464259 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.464458 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.464458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.464459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.464554 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:46.464602 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:46.464853 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 05:05:46.464921 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:05:46.465044 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.457178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38981 (local address 127.14.33.2:38991)
0925 05:05:46.457343 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:46.457400 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:46.457888 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:46.458103 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:46.458106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:46.458690 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:46.458870 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:46.458875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.459540 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.459540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:46.460298 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:46.460300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.460641 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.460641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:46.460731 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.461223 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:46.461230 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:46.462533 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:46.463938 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:46.463943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:46.463945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:46.464146 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:46.464315 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:46.464316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:46.464316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:46.464406 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:46.464607 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:46.464611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:46.464714 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 05:05:46.464793 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:05:46.464900 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260925 05:05:46.465553 17215 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38981
I20260925 05:05:46.465664 17215 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:46.465731 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17086
I20260925 05:05:46.465834 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:46.465905 17215 heartbeater.cc:511] Master 127.14.33.62:38981 requested a full tablet report, sending...
I20260925 05:05:46.466037 14468 external_mini_cluster.cc:1428] Setting key 70c60a592298fc0ef682a9915048f747
I20260925 05:05:46.466284 16890 ts_manager.cc:194] Registered new tserver with Master: cef1d492236843559ee921d14a305332 (127.14.33.2:40881)
I20260925 05:05:46.466784 16890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:38991
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:46.488780 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38981
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:46.565855 17220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:46.566057 17220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:46.566079 17220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:46.567065 17220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:46.567097 17220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:46.567121 17220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:46.568300 17220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:46.568445 17220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:46.568584 17220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:46.572851 17225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.572824 17228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:46.572870 17226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:46.573303 17220 server_base.cc:1034] running on GCE node
I20260925 05:05:46.573575 17220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:46.573777 17220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:46.574914 17220 hybrid_clock.cc:648] HybridClock initialized: now 1790312746574904 us; error 31 us; skew 500 ppm
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:46.577054 17220 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:46.577497 17220 webserver.cc:460] Webserver started at http://127.14.33.3:39763/ using document root <none> and password file <none>
I20260925 05:05:46.577704 17220 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:46.577747 17220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:46.577877 17220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:46.578831 17220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "aeda61d3b20a481c8d92771e359029dd"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "5d2b497bfabec7e79c3581401763b716"
server_key_iv: "c2d804cbbef44f8ed189bcdb2819e45e"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.579191 17220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "aeda61d3b20a481c8d92771e359029dd"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "5d2b497bfabec7e79c3581401763b716"
server_key_iv: "c2d804cbbef44f8ed189bcdb2819e45e"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.580467 17220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:46.581324 17235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.581533 17220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:46.581630 17220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "aeda61d3b20a481c8d92771e359029dd"
format_stamp: "Formatted at 2026-09-25 05:05:46 on dist-test-slave-4dmq"
server_key: "5d2b497bfabec7e79c3581401763b716"
server_key_iv: "c2d804cbbef44f8ed189bcdb2819e45e"
server_key_version: "encryptionkey@0"
I20260925 05:05:46.581719 17220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:46.592495 17220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:46.594487 17220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:46.594671 17220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:46.594892 17220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:46.595197 17220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:46.595255 17220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.595299 17220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:46.595333 17220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:46.601619 17220 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:36019
I20260925 05:05:46.601691 17348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:36019 every 8 connection(s)
I20260925 05:05:46.601966 17220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:46.602953 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17220
I20260925 05:05:46.603067 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:46.603250 14468 external_mini_cluster.cc:1428] Setting key 77016351d094edcdb61fab6a3d499d3c
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:46.610730 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.602737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:60017 (local address 127.14.33.62:38981)
0925 05:05:46.602800 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:46.602801 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:46.602991 (+ 190us) server_negotiation.cc:407] Connection header received
0925 05:05:46.603623 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:46.603624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:46.603635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:46.603692 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:46.604542 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.605038 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.606014 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.606227 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.606295 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.608169 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:46.608170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:46.608171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:46.608177 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:46.609413 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.609823 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.609825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.609825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.609889 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:46.610140 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:46.610141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:46.610142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:46.610248 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:46.610321 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:46.610569 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 05:05:46.610643 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:05:46.610890 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:46.602709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38981 (local address 127.14.33.3:60017)
0925 05:05:46.602876 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:46.602915 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:46.603514 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:46.603758 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:46.603761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:46.604287 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:46.604448 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:46.604452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.605156 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.605157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:46.605916 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:46.605918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:46.606299 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:46.606300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:46.606417 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:46.606853 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:46.606860 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:46.608071 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:46.609505 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:46.609512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:46.609514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:46.609727 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:46.609954 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:46.609955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:46.609955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:46.610061 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:46.610313 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:46.610319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:46.610447 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 05:05:46.610549 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:05:46.610687 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260925 05:05:46.611353 17349 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38981
I20260925 05:05:46.611421 17349 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:46.611594 17349 heartbeater.cc:511] Master 127.14.33.62:38981 requested a full tablet report, sending...
I20260925 05:05:46.611941 16890 ts_manager.cc:194] Registered new tserver with Master: aeda61d3b20a481c8d92771e359029dd (127.14.33.3:36019)
I20260925 05:05:46.612349 16890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:60017
I20260925 05:05:46.614820 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:05:46.625993 16929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:46.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.14.33.62:38981)
0925 05:05:46.625195 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:46.625197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:46.625369 (+ 172us) server_negotiation.cc:407] Connection header received
0925 05:05:46.625421 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:46.625422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:46.625430 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:46.625466 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:46.625938 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59484: BlockingRecv error: recv got EOF from 127.0.0.1:59484 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 05:05:46.626861 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 16952
I20260925 05:05:46.631141 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17086
I20260925 05:05:46.635411 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17220
I20260925 05:05:46.639326 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 16858
2026-09-25T05:05:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4800 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[17361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[17361](info): set up 2 sockets
Sep 25 05:05:46 dist-test-slave-4dmq krb5kdc[17361](info): commencing operation
krb5kdc: starting...
W20260925 05:05:47.658083 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:47 dist-test-slave-4dmq krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:47.765755 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40461
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:47.844867 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:47.845109 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:47.845192 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:47.846323 17374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:47.846390 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:47.846418 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:47.846450 17374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:47.846482 17374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:47.848006 17374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:47.848277 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:47.848503 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:47.853247 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:47.853255 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:47.853413 17374 server_base.cc:1034] running on GCE node
W20260925 05:05:47.853310 17380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:47.853727 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:47.853976 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:47.855139 17374 hybrid_clock.cc:648] HybridClock initialized: now 1790312747855135 us; error 35 us; skew 500 ppm
Sep 25 05:05:47 dist-test-slave-4dmq krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312747, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:47.861815 17374 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:47.862229 17374 webserver.cc:460] Webserver started at http://127.14.33.62:36619/ using document root <none> and password file <none>
I20260925 05:05:47.862452 17374 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:47.862516 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:47.862660 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:47.863699 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "1f021f9644db42d69e4c2f693785ca64"
format_stamp: "Formatted at 2026-09-25 05:05:47 on dist-test-slave-4dmq"
server_key: "cb7a90b3785a309788f8fefea8d878c6"
server_key_iv: "9cac508a125d5b85b418e5d79a5b0bea"
server_key_version: "encryptionkey@0"
I20260925 05:05:47.864064 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "1f021f9644db42d69e4c2f693785ca64"
format_stamp: "Formatted at 2026-09-25 05:05:47 on dist-test-slave-4dmq"
server_key: "cb7a90b3785a309788f8fefea8d878c6"
server_key_iv: "9cac508a125d5b85b418e5d79a5b0bea"
server_key_version: "encryptionkey@0"
I20260925 05:05:47.865402 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:47.866094 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:47.866277 17374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:47.866379 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "1f021f9644db42d69e4c2f693785ca64"
format_stamp: "Formatted at 2026-09-25 05:05:47 on dist-test-slave-4dmq"
server_key: "cb7a90b3785a309788f8fefea8d878c6"
server_key_iv: "9cac508a125d5b85b418e5d79a5b0bea"
server_key_version: "encryptionkey@0"
I20260925 05:05:47.866449 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:47.874835 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:47.876469 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:47.876611 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:47.880445 17374 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40461
I20260925 05:05:47.880474 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40461 every 8 connection(s)
I20260925 05:05:47.880927 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:47.881629 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:47.883761 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64: Bootstrap starting.
I20260925 05:05:47.884341 17442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:47.884572 17442 log.cc:824] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:47.885267 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64: No bootstrap required, opened a new log
I20260925 05:05:47.886758 17442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f021f9644db42d69e4c2f693785ca64" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40461 } }
I20260925 05:05:47.886881 17442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:47.886904 17442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f021f9644db42d69e4c2f693785ca64, State: Initialized, Role: FOLLOWER
I20260925 05:05:47.887045 17442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f021f9644db42d69e4c2f693785ca64" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40461 } }
I20260925 05:05:47.887121 17442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:47.887143 17442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:47.887171 17442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:47.887965 17442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f021f9644db42d69e4c2f693785ca64" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40461 } }
I20260925 05:05:47.888056 17442 leader_election.cc:302] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f021f9644db42d69e4c2f693785ca64; no voters:
I20260925 05:05:47.888172 17442 leader_election.cc:288] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:47.888237 17445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:47.888392 17445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [term 1 LEADER]: Becoming Leader. State: Replica: 1f021f9644db42d69e4c2f693785ca64, State: Running, Role: LEADER
I20260925 05:05:47.888404 17442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:47.888554 17445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f021f9644db42d69e4c2f693785ca64" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40461 } }
I20260925 05:05:47.888890 17446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f021f9644db42d69e4c2f693785ca64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f021f9644db42d69e4c2f693785ca64" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40461 } } }
I20260925 05:05:47.888892 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f021f9644db42d69e4c2f693785ca64. Latest consensus state: current_term: 1 leader_uuid: "1f021f9644db42d69e4c2f693785ca64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f021f9644db42d69e4c2f693785ca64" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40461 } } }
I20260925 05:05:47.888993 17446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:47.888998 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:47.889498 17454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:47.890022 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17374
I20260925 05:05:47.890112 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:47.890278 14468 external_mini_cluster.cc:1428] Setting key e150ba9952701abda2d2d4d482f252ec
I20260925 05:05:47.890592 17454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:47.891964 17454 catalog_manager.cc:1347] Generated new cluster ID: a30891b19f6d48aaa9ed34e32283e7d4
I20260925 05:05:47.892014 17454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:05:47 dist-test-slave-4dmq krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:47.901678 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:47.890687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.14.33.62:40461)
0925 05:05:47.891188 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:47.891207 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:47.891398 (+ 191us) server_negotiation.cc:407] Connection header received
0925 05:05:47.891518 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:47.891525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:47.891585 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:47.891728 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:47.892463 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:47.893902 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:47.894918 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:47.895637 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:47.895745 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:47.898291 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:47.898295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:47.898295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:47.898305 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:47.900138 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:47.900514 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:47.900516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:47.900516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:47.900621 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:47.900824 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:47.900825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:47.900825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:47.901104 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:47.901270 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:47.901363 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 05:05:47.901480 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":445,"thread_start_us":72,"threads_started":1}
I20260925 05:05:47.902592 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:47.910008 17454 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:47.910559 17454 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:47.915555 17454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f021f9644db42d69e4c2f693785ca64: Generated new TSK 0
I20260925 05:05:47.915723 17454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:47.917865 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:47.914065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.14.33.62:40461)
0925 05:05:47.914100 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:47.914102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:47.914334 (+ 232us) server_negotiation.cc:407] Connection header received
0925 05:05:47.914380 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:47.914381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:47.914392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:47.914457 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:47.914848 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:47.915604 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:47.916396 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:47.916712 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:47.916780 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:47.917435 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:47.917436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:47.917436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:47.917635 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:47.917652 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:47.917658 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:47.917721 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:47.917769 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 05:05:47.917792 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:05:47.920127 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17374
2026-09-25T05:05:47Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1277 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:47 dist-test-slave-4dmq krb5kdc[17475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:47 dist-test-slave-4dmq krb5kdc[17475](info): set up 2 sockets
Sep 25 05:05:47 dist-test-slave-4dmq krb5kdc[17475](info): commencing operation
krb5kdc: starting...
W20260925 05:05:48.926061 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:49.032590 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:44287
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:49.107100 17488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:49.107301 17488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:49.107335 17488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:49.108364 17488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:49.108421 17488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:49.108443 17488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:49.108467 17488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:49.108495 17488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:49.109910 17488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:44287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:49.110188 17488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:49.110410 17488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:49.114799 17493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.114768 17494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.114766 17496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:49.114790 17488 server_base.cc:1034] running on GCE node
I20260925 05:05:49.115191 17488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:49.115516 17488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:49.116707 17488 hybrid_clock.cc:648] HybridClock initialized: now 1790312749116672 us; error 37 us; skew 500 ppm
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:49.119166 17488 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:49.119626 17488 webserver.cc:460] Webserver started at http://127.14.33.62:38997/ using document root <none> and password file <none>
I20260925 05:05:49.119855 17488 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:49.119920 17488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:49.120061 17488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:49.120997 17488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "efac0095edfc4cb98e2824a3f0bad52e"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "49fbb7332a6d8e1a5d58561d152363b6"
server_key_iv: "a729da657552b3345180e69ae3623a50"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.121361 17488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "efac0095edfc4cb98e2824a3f0bad52e"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "49fbb7332a6d8e1a5d58561d152363b6"
server_key_iv: "a729da657552b3345180e69ae3623a50"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.122587 17488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:49.123258 17503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.123425 17488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:49.123518 17488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "efac0095edfc4cb98e2824a3f0bad52e"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "49fbb7332a6d8e1a5d58561d152363b6"
server_key_iv: "a729da657552b3345180e69ae3623a50"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.123594 17488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:49.146651 17488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:49.148826 17488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:49.148979 17488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:49.152551 17488 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:44287
I20260925 05:05:49.152583 17555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:44287 every 8 connection(s)
I20260925 05:05:49.152932 17488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:49.153836 17556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:49.155896 17556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e: Bootstrap starting.
I20260925 05:05:49.156374 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17488
I20260925 05:05:49.156404 17556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:49.156482 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:49.156661 14468 external_mini_cluster.cc:1428] Setting key 63d19d190047a43077727c373f09499c
I20260925 05:05:49.156697 17556 log.cc:824] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:49.157433 17556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e: No bootstrap required, opened a new log
I20260925 05:05:49.159322 17556 raft_consensus.cc:348] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efac0095edfc4cb98e2824a3f0bad52e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 44287 } }
I20260925 05:05:49.159442 17556 raft_consensus.cc:374] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:49.159463 17556 raft_consensus.cc:729] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efac0095edfc4cb98e2824a3f0bad52e, State: Initialized, Role: FOLLOWER
I20260925 05:05:49.159631 17556 consensus_queue.cc:260] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efac0095edfc4cb98e2824a3f0bad52e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 44287 } }
I20260925 05:05:49.159724 17556 raft_consensus.cc:388] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:49.159776 17556 raft_consensus.cc:482] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:49.159842 17556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:49.160342 17556 raft_consensus.cc:504] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efac0095edfc4cb98e2824a3f0bad52e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 44287 } }
I20260925 05:05:49.160454 17556 leader_election.cc:302] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efac0095edfc4cb98e2824a3f0bad52e; no voters:
I20260925 05:05:49.160629 17556 leader_election.cc:288] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:49.160646 17561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:49.160753 17561 raft_consensus.cc:686] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [term 1 LEADER]: Becoming Leader. State: Replica: efac0095edfc4cb98e2824a3f0bad52e, State: Running, Role: LEADER
I20260925 05:05:49.160889 17556 sys_catalog.cc:564] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:49.160910 17561 consensus_queue.cc:237] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efac0095edfc4cb98e2824a3f0bad52e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 44287 } }
I20260925 05:05:49.161322 17563 sys_catalog.cc:455] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [sys.catalog]: SysCatalogTable state changed. Reason: New leader efac0095edfc4cb98e2824a3f0bad52e. Latest consensus state: current_term: 1 leader_uuid: "efac0095edfc4cb98e2824a3f0bad52e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efac0095edfc4cb98e2824a3f0bad52e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 44287 } } }
I20260925 05:05:49.161414 17563 sys_catalog.cc:458] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:49.161605 17562 sys_catalog.cc:455] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efac0095edfc4cb98e2824a3f0bad52e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efac0095edfc4cb98e2824a3f0bad52e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 44287 } } }
I20260925 05:05:49.161856 17562 sys_catalog.cc:458] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:49.162009 17565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:49.162467 17565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:49.164189 17565 catalog_manager.cc:1347] Generated new cluster ID: cfb71c4549ba46e59ac8207cbcbed707
I20260925 05:05:49.164242 17565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:49.166823 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.157082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.14.33.62:44287)
0925 05:05:49.157241 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:49.157250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:49.157281 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:05:49.157770 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:49.157777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:49.157823 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:49.157979 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:49.158581 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.159434 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.160281 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.160837 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.161211 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.163562 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:49.163566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:49.163566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:49.163576 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:49.165278 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.165666 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.165668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.165669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.165765 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.165960 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.165961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.165961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.166235 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:49.166440 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:49.166505 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:05:49.166630 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260925 05:05:49.170701 17565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:49.171164 17565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:49.179728 17565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efac0095edfc4cb98e2824a3f0bad52e: Generated new TSK 0
I20260925 05:05:49.179888 17565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:49.203507 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:44287
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:49.280153 17582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:49.280381 17582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:49.280417 17582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:49.281538 17582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:49.281601 17582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:49.281644 17582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:49.283100 17582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:44287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:49.283361 17582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:49.283581 17582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:49.288483 17590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.288493 17588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:49.288591 17582 server_base.cc:1034] running on GCE node
W20260925 05:05:49.288552 17587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:49.288827 17582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:49.289052 17582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:49.290206 17582 hybrid_clock.cc:648] HybridClock initialized: now 1790312749290195 us; error 28 us; skew 500 ppm
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:49.292601 17582 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:49.293057 17582 webserver.cc:460] Webserver started at http://127.14.33.1:35049/ using document root <none> and password file <none>
I20260925 05:05:49.293318 17582 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:49.293385 17582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:49.293524 17582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:49.294520 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "250b8507c8484d34bde64bf26ba19f65"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "3e069e382eec982b72e61e1658927a47"
server_key_iv: "34fcb85d9c5fee57b232a5cc96d45b44"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.294852 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "250b8507c8484d34bde64bf26ba19f65"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "3e069e382eec982b72e61e1658927a47"
server_key_iv: "34fcb85d9c5fee57b232a5cc96d45b44"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.296053 17582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:49.296717 17597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.296881 17582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:49.296972 17582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "250b8507c8484d34bde64bf26ba19f65"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "3e069e382eec982b72e61e1658927a47"
server_key_iv: "34fcb85d9c5fee57b232a5cc96d45b44"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.297042 17582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:49.323256 17582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:49.325008 17582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:49.325179 17582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:49.325402 17582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:49.325666 17582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:49.325712 17582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.325748 17582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:49.325779 17582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.331557 17582 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39253
I20260925 05:05:49.331607 17710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39253 every 8 connection(s)
I20260925 05:05:49.331864 17582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:49.338155 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17582
I20260925 05:05:49.338234 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:49.338402 14468 external_mini_cluster.cc:1428] Setting key 142cb41204c6b20158cc343c72b8506d
I20260925 05:05:49.340325 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.332515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:42733 (local address 127.14.33.62:44287)
0925 05:05:49.332599 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:49.332602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:49.332752 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:05:49.333236 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:49.333237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:49.333248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:49.333311 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:49.334099 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.334615 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.335460 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.335695 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.335757 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.337642 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:49.337643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:49.337644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:49.337651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:49.339017 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.339424 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.339425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.339426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.339483 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.339725 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.339726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.339727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.339840 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:49.339909 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:49.340161 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 05:05:49.340238 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:05:49.340497 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.332453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:44287 (local address 127.14.33.1:42733)
0925 05:05:49.332634 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:49.332671 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:49.333101 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:49.333358 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:49.333360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:49.333856 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:49.334022 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:49.334026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.334695 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.334696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:49.335389 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:49.335390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.335773 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.335774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:49.335868 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.336278 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:49.336285 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:49.337549 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:49.339118 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:49.339125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:49.339127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:49.339347 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:49.339555 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:49.339556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:49.339556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:49.339655 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:49.339912 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:49.339919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:49.340040 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 05:05:49.340138 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:05:49.340290 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260925 05:05:49.340969 17711 heartbeater.cc:347] Connected to a master server at 127.14.33.62:44287
I20260925 05:05:49.341069 17711 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:49.341250 17711 heartbeater.cc:511] Master 127.14.33.62:44287 requested a full tablet report, sending...
I20260925 05:05:49.341761 17520 ts_manager.cc:194] Registered new tserver with Master: 250b8507c8484d34bde64bf26ba19f65 (127.14.33.1:39253)
I20260925 05:05:49.342346 17520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:42733
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:49.359936 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:44287
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:49.435437 17716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:49.435597 17716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:49.435617 17716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:49.436650 17716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:49.436686 17716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:49.436710 17716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:49.438067 17716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:44287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:49.438304 17716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:49.438494 17716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:05:49.443320 17716 server_base.cc:1034] running on GCE node
W20260925 05:05:49.443262 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.443257 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.443290 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:49.443729 17716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:49.444024 17716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:49.445173 17716 hybrid_clock.cc:648] HybridClock initialized: now 1790312749445161 us; error 42 us; skew 500 ppm
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:49.447373 17716 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:49.447818 17716 webserver.cc:460] Webserver started at http://127.14.33.2:37513/ using document root <none> and password file <none>
I20260925 05:05:49.448037 17716 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:49.448099 17716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:49.448220 17716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:49.449177 17716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "1ad458801add4a59853f094a9c1bde86"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "f3782ff0b7074bd31fda9045aafb0fed"
server_key_iv: "ab8f85ba4a7e3e9f852f9de7ed71b5a9"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.449589 17716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "1ad458801add4a59853f094a9c1bde86"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "f3782ff0b7074bd31fda9045aafb0fed"
server_key_iv: "ab8f85ba4a7e3e9f852f9de7ed71b5a9"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.451422 17716 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 05:05:49.452346 17731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.452556 17716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:49.452636 17716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "1ad458801add4a59853f094a9c1bde86"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "f3782ff0b7074bd31fda9045aafb0fed"
server_key_iv: "ab8f85ba4a7e3e9f852f9de7ed71b5a9"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.452709 17716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:49.465731 17716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:49.467794 17716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:49.467954 17716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:49.468152 17716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:49.468443 17716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:49.468492 17716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.468526 17716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:49.468552 17716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.473583 17716 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:35411
I20260925 05:05:49.473647 17844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:35411 every 8 connection(s)
I20260925 05:05:49.473932 17716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:49.482836 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.474549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:51103 (local address 127.14.33.62:44287)
0925 05:05:49.474631 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:49.474634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:49.474797 (+ 163us) server_negotiation.cc:407] Connection header received
0925 05:05:49.475346 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:49.475348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:49.475360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:49.475419 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:49.476519 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.477019 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.477987 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.478217 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.478296 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.480216 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:49.480217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:49.480218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:49.480225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:49.481570 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.481948 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.481950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.481950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.482007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.482236 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.482238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.482238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.482345 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:49.482415 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:49.482658 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 05:05:49.482732 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:05:49.483000 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.474518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:44287 (local address 127.14.33.2:51103)
0925 05:05:49.474670 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:49.474716 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:49.475254 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:49.475481 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:49.475484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:49.476242 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:49.476409 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:49.476413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.477107 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.477108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:49.477894 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:49.477896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.478299 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.478300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:49.478423 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.478874 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:49.478882 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:49.480105 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:49.481663 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:49.481670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:49.481672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:49.481876 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:49.482078 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:49.482078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:49.482079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:49.482175 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:49.482407 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:49.482412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:49.482537 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 05:05:49.482627 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 05:05:49.482784 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 05:05:49.483417 17845 heartbeater.cc:347] Connected to a master server at 127.14.33.62:44287
I20260925 05:05:49.483490 17845 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:49.483659 17845 heartbeater.cc:511] Master 127.14.33.62:44287 requested a full tablet report, sending...
I20260925 05:05:49.484045 17520 ts_manager.cc:194] Registered new tserver with Master: 1ad458801add4a59853f094a9c1bde86 (127.14.33.2:35411)
I20260925 05:05:49.484094 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17716
I20260925 05:05:49.484187 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:49.484365 14468 external_mini_cluster.cc:1428] Setting key d95205da9d2d61f935f0ba6f80d125c7
I20260925 05:05:49.484514 17520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:51103
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:49.505656 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:44287
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:49.582722 17850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:49.582907 17850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:49.582929 17850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:49.583997 17850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:49.584033 17850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:49.584056 17850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:49.585397 17850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:44287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:49.585569 17850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:49.585721 17850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:49.590354 17856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.590354 17858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:49.590591 17855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:49.590585 17850 server_base.cc:1034] running on GCE node
I20260925 05:05:49.590802 17850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:49.590981 17850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:49.592119 17850 hybrid_clock.cc:648] HybridClock initialized: now 1790312749592107 us; error 27 us; skew 500 ppm
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:49.594527 17850 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:49.594884 17850 webserver.cc:460] Webserver started at http://127.14.33.3:34981/ using document root <none> and password file <none>
I20260925 05:05:49.595041 17850 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:49.595077 17850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:49.595167 17850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:49.596066 17850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "a73fa7ab9cac4719b302295823f40150"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "158e649d9d499cee28ac90683ee4f8c6"
server_key_iv: "744cba02faef7cac8b1862e7d8b64377"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.596361 17850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "a73fa7ab9cac4719b302295823f40150"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "158e649d9d499cee28ac90683ee4f8c6"
server_key_iv: "744cba02faef7cac8b1862e7d8b64377"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.597620 17850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:49.598266 17865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.598454 17850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:49.598546 17850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "a73fa7ab9cac4719b302295823f40150"
format_stamp: "Formatted at 2026-09-25 05:05:49 on dist-test-slave-4dmq"
server_key: "158e649d9d499cee28ac90683ee4f8c6"
server_key_iv: "744cba02faef7cac8b1862e7d8b64377"
server_key_version: "encryptionkey@0"
I20260925 05:05:49.598613 17850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:49.610906 17850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:49.612555 17850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:49.612653 17850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:49.612804 17850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:49.613018 17850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:49.613045 17850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.613065 17850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:49.613076 17850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:49.618549 17850 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:43131
I20260925 05:05:49.618603 17978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:43131 every 8 connection(s)
I20260925 05:05:49.618896 17850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:49.619527 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 17850
I20260925 05:05:49.619633 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:49.619801 14468 external_mini_cluster.cc:1428] Setting key 3fa44eb7b763b6c40286ba4214ced2ec
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:49.627718 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.619521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:43551 (local address 127.14.33.62:44287)
0925 05:05:49.619586 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:49.619589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:49.619863 (+ 274us) server_negotiation.cc:407] Connection header received
0925 05:05:49.620460 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:49.620461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:49.620473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:49.620513 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:49.621410 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.621865 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.622838 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.623026 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.623100 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.625023 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:49.625025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:49.625026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:49.625033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:49.626413 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.626788 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.626789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.626790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.626846 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.627083 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.627084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.627085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.627218 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:49.627293 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:49.627555 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 05:05:49.627630 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:05:49.627892 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.619493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:44287 (local address 127.14.33.3:43551)
0925 05:05:49.619752 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:49.619792 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:49.620346 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:49.620583 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:49.620586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:49.621077 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:49.621323 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:49.621327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.621974 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.621975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:49.622745 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:49.622747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.623104 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.623105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:49.623244 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.623721 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:49.623728 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:49.624934 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:49.626485 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:49.626491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:49.626493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:49.626701 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:49.626910 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:49.626911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:49.626912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:49.627013 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:49.627283 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:49.627289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:49.627431 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 05:05:49.627533 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 05:05:49.627669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260925 05:05:49.628314 17979 heartbeater.cc:347] Connected to a master server at 127.14.33.62:44287
I20260925 05:05:49.628379 17979 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:49.628548 17979 heartbeater.cc:511] Master 127.14.33.62:44287 requested a full tablet report, sending...
I20260925 05:05:49.628911 17519 ts_manager.cc:194] Registered new tserver with Master: a73fa7ab9cac4719b302295823f40150 (127.14.33.3:43131)
I20260925 05:05:49.629336 17519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:43551
I20260925 05:05:49.631367 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312749, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:49.652189 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:49.645411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.14.33.62:44287)
0925 05:05:49.645485 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:49.645487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:49.645582 (+ 95us) server_negotiation.cc:407] Connection header received
0925 05:05:49.645644 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:49.645645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:49.645654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:49.645703 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:49.646217 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.646688 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.647439 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:49.647622 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:49.647712 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:49.649903 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:49.649904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:49.649904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:49.649910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:49.651285 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.651563 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.651564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.651564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.651610 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:49.651751 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:49.651752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:49.651752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:49.651872 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:49.651940 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:49.652026 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:05:49.652080 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 05:05:49.652400 17519 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54506
I20260925 05:05:49.652756 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17582
I20260925 05:05:49.657518 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17716
I20260925 05:05:49.662436 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17850
I20260925 05:05:49.667014 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 17488
2026-09-25T05:05:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1749 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17991](info): set up 2 sockets
Sep 25 05:05:49 dist-test-slave-4dmq krb5kdc[17991](info): commencing operation
krb5kdc: starting...
W20260925 05:05:50.657869 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:50 dist-test-slave-4dmq krb5kdc[17991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:50.761878 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36881
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:50.836934 18004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:50.837109 18004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:50.837189 18004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:50.838297 18004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:50.838333 18004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:50.838348 18004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:50.838361 18004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:50.838372 18004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:50.839798 18004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:50.839968 18004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:50.840118 18004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:50.844601 18010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:50.844592 18009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:50.844588 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:50.844836 18004 server_base.cc:1034] running on GCE node
I20260925 05:05:50.845067 18004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:50.845367 18004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:50.846540 18004 hybrid_clock.cc:648] HybridClock initialized: now 1790312750846531 us; error 45 us; skew 500 ppm
Sep 25 05:05:50 dist-test-slave-4dmq krb5kdc[17991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312750, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:50.848780 18004 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:50.849211 18004 webserver.cc:460] Webserver started at http://127.14.33.62:43357/ using document root <none> and password file <none>
I20260925 05:05:50.849447 18004 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:50.849510 18004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:50.849648 18004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:50.850580 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "901d86896dcc4cc59a24100c550261ca"
format_stamp: "Formatted at 2026-09-25 05:05:50 on dist-test-slave-4dmq"
server_key: "d7181a89da3bea4ff1c7c4558272b1fe"
server_key_iv: "452f8a8023d698196501041c8a8e424a"
server_key_version: "encryptionkey@0"
I20260925 05:05:50.850921 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "901d86896dcc4cc59a24100c550261ca"
format_stamp: "Formatted at 2026-09-25 05:05:50 on dist-test-slave-4dmq"
server_key: "d7181a89da3bea4ff1c7c4558272b1fe"
server_key_iv: "452f8a8023d698196501041c8a8e424a"
server_key_version: "encryptionkey@0"
I20260925 05:05:50.852173 18004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:50.852840 18019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:50.853006 18004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:50.853091 18004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "901d86896dcc4cc59a24100c550261ca"
format_stamp: "Formatted at 2026-09-25 05:05:50 on dist-test-slave-4dmq"
server_key: "d7181a89da3bea4ff1c7c4558272b1fe"
server_key_iv: "452f8a8023d698196501041c8a8e424a"
server_key_version: "encryptionkey@0"
I20260925 05:05:50.853175 18004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:50.859882 18004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:50.861778 18004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:50.861922 18004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:50.865708 18004 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:36881
I20260925 05:05:50.865746 18071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:36881 every 8 connection(s)
I20260925 05:05:50.866199 18004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:50.866870 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:50.869025 18072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca: Bootstrap starting.
I20260925 05:05:50.869648 18072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:50.869958 18072 log.cc:824] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:50.870606 18072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca: No bootstrap required, opened a new log
I20260925 05:05:50.872148 18072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d86896dcc4cc59a24100c550261ca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36881 } }
I20260925 05:05:50.872284 18072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:50.872341 18072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901d86896dcc4cc59a24100c550261ca, State: Initialized, Role: FOLLOWER
I20260925 05:05:50.872485 18072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d86896dcc4cc59a24100c550261ca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36881 } }
I20260925 05:05:50.872543 18072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:50.872563 18072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:50.872633 18072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:50.873606 18072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d86896dcc4cc59a24100c550261ca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36881 } }
I20260925 05:05:50.873703 18072 leader_election.cc:302] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 901d86896dcc4cc59a24100c550261ca; no voters:
I20260925 05:05:50.873901 18072 leader_election.cc:288] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:50.873974 18075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:50.874150 18072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:50.874169 18075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [term 1 LEADER]: Becoming Leader. State: Replica: 901d86896dcc4cc59a24100c550261ca, State: Running, Role: LEADER
I20260925 05:05:50.874297 18075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d86896dcc4cc59a24100c550261ca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36881 } }
I20260925 05:05:50.874665 18076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "901d86896dcc4cc59a24100c550261ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d86896dcc4cc59a24100c550261ca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36881 } } }
I20260925 05:05:50.874884 18076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:50.874799 18077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 901d86896dcc4cc59a24100c550261ca. Latest consensus state: current_term: 1 leader_uuid: "901d86896dcc4cc59a24100c550261ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d86896dcc4cc59a24100c550261ca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36881 } } }
I20260925 05:05:50.875156 18077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:50.875232 18084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:50.875711 18084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:50.875938 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18004
I20260925 05:05:50.876047 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:50.876209 14468 external_mini_cluster.cc:1428] Setting key fd3230a3f011c065dbedee7fa8589bd4
I20260925 05:05:50.877492 18084 catalog_manager.cc:1347] Generated new cluster ID: 1e45cf0dccdf433e9f4600bca2b6ced7
I20260925 05:05:50.877542 18084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:05:50 dist-test-slave-4dmq krb5kdc[17991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:50.887138 18084 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:50.887599 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:50.876650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.14.33.62:36881)
0925 05:05:50.876816 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:50.876825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:50.876951 (+ 126us) server_negotiation.cc:407] Connection header received
0925 05:05:50.877079 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:50.877086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:50.877161 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:50.878084 (+ 923us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:50.878173 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:50.879595 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:50.880592 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:50.881256 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:50.881358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:50.884000 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:50.884004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:50.884005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:50.884017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:50.885933 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:50.886294 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:50.886297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:50.886297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:50.886406 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:50.886611 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:50.886613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:50.886613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:50.886872 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:50.887049 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:50.887224 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 05:05:50.887373 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260925 05:05:50.887804 18084 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:50.891680 18084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 901d86896dcc4cc59a24100c550261ca: Generated new TSK 0
I20260925 05:05:50.891858 18084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:50.919456 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36881
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:50.997169 18098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:50.997354 18098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:50.997375 18098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:50.998471 18098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:50.998510 18098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:50.998534 18098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:50.999980 18098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:51.000146 18098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:51.000365 18098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:51.005201 18103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:51.005259 18106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:51.005439 18098 server_base.cc:1034] running on GCE node
W20260925 05:05:51.005200 18104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:51.005716 18098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:51.005923 18098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:51.007076 18098 hybrid_clock.cc:648] HybridClock initialized: now 1790312751007052 us; error 37 us; skew 500 ppm
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:51.009588 18098 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:51.010027 18098 webserver.cc:460] Webserver started at http://127.14.33.1:36837/ using document root <none> and password file <none>
I20260925 05:05:51.010236 18098 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:51.010299 18098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:51.010470 18098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:51.011423 18098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "14c98c0d5e444a9fb8918006f3c8cda9"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "1686f7e2fd909987f41d94f90d3cc1b0"
server_key_iv: "fed99a41b79644741dec9a799f419b63"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.011803 18098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "14c98c0d5e444a9fb8918006f3c8cda9"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "1686f7e2fd909987f41d94f90d3cc1b0"
server_key_iv: "fed99a41b79644741dec9a799f419b63"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.013178 18098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:05:51.013898 18113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.014066 18098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:51.014171 18098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "14c98c0d5e444a9fb8918006f3c8cda9"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "1686f7e2fd909987f41d94f90d3cc1b0"
server_key_iv: "fed99a41b79644741dec9a799f419b63"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.014251 18098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:51.025013 18098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:51.026841 18098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:51.027015 18098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:51.027223 18098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:51.027508 18098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:51.027558 18098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.027598 18098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:51.027642 18098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.033358 18098 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39195
I20260925 05:05:51.033434 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39195 every 8 connection(s)
I20260925 05:05:51.033727 18098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:51.042346 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.034397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:33627 (local address 127.14.33.62:36881)
0925 05:05:51.034477 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:51.034480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:51.034642 (+ 162us) server_negotiation.cc:407] Connection header received
0925 05:05:51.035470 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:51.035471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:51.035486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:51.035537 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:51.036058 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.036536 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.037522 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.037720 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.037774 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.039711 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:51.039712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:51.039712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:51.039720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:51.041168 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.041553 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.041554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.041555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.041607 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.041815 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.041815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.041816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.041915 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:51.041965 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:51.042207 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 05:05:51.042274 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:05:51.042493 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.034368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36881 (local address 127.14.33.1:33627)
0925 05:05:51.034528 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:51.034560 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:51.035383 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:51.035585 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:51.035587 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:51.035817 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:51.035987 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:51.035991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.036610 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.036611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:51.037446 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:51.037449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.037797 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.037798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:51.037885 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.038299 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:51.038306 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:51.039620 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:51.041258 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:51.041264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:51.041266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:51.041489 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:51.041665 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:51.041666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:51.041666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:51.041755 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:51.041970 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:51.041974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:51.042095 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 05:05:51.042174 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:05:51.042283 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260925 05:05:51.043049 18227 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36881
I20260925 05:05:51.043149 18227 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:51.043319 18227 heartbeater.cc:511] Master 127.14.33.62:36881 requested a full tablet report, sending...
I20260925 05:05:51.043702 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18098
I20260925 05:05:51.043829 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:51.043939 18036 ts_manager.cc:194] Registered new tserver with Master: 14c98c0d5e444a9fb8918006f3c8cda9 (127.14.33.1:39195)
I20260925 05:05:51.044031 14468 external_mini_cluster.cc:1428] Setting key 3cacddc8d7bab3adde37bed32716eb9a
I20260925 05:05:51.044680 18036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:33627
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:51.067394 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36881
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:51.145777 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:51.145964 18232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:51.145985 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:51.147045 18232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:51.147080 18232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:51.147106 18232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:51.148509 18232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:51.148685 18232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:51.148838 18232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:51.153404 18238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:51.153401 18240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:51.153645 18232 server_base.cc:1034] running on GCE node
W20260925 05:05:51.153404 18237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:51.153946 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:51.154227 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:51.155395 18232 hybrid_clock.cc:648] HybridClock initialized: now 1790312751155381 us; error 26 us; skew 500 ppm
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:51.157763 18232 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:51.158211 18232 webserver.cc:460] Webserver started at http://127.14.33.2:36299/ using document root <none> and password file <none>
I20260925 05:05:51.158417 18232 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:51.158480 18232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:51.158622 18232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:51.159586 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "843a3057a8de40dcbcb96e382b5222c4"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "4bb8dfa9be04dde209e15e321813ca22"
server_key_iv: "97e7ef269ccb31d43fffdb302b6431d6"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.159952 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "843a3057a8de40dcbcb96e382b5222c4"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "4bb8dfa9be04dde209e15e321813ca22"
server_key_iv: "97e7ef269ccb31d43fffdb302b6431d6"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.161217 18232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:51.161918 18247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.162073 18232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:51.162164 18232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "843a3057a8de40dcbcb96e382b5222c4"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "4bb8dfa9be04dde209e15e321813ca22"
server_key_iv: "97e7ef269ccb31d43fffdb302b6431d6"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.162256 18232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:51.170653 18232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:51.172248 18232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:51.172396 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:51.172600 18232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:51.172866 18232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:51.172919 18232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.172955 18232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:51.172982 18232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.178855 18232 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:35947
I20260925 05:05:51.178905 18360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:35947 every 8 connection(s)
I20260925 05:05:51.179175 18232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:51.182070 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18232
I20260925 05:05:51.182204 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:51.182386 14468 external_mini_cluster.cc:1428] Setting key 6192f583942ef7c823cb74183239e008
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:51.188748 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.179794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36881 (local address 127.14.33.2:56267)
0925 05:05:51.179927 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:51.179964 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:51.180799 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:51.180991 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:51.180994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:51.181318 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:51.181504 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:51.181508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.182148 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.182149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:51.183351 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:51.183354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.183717 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.183718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:51.183847 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.184315 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:51.184325 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:51.185726 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:51.187550 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:51.187555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:51.187556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:51.187764 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:51.187956 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:51.187957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:51.187957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:51.188040 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:51.188325 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:51.188329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:51.188428 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 05:05:51.188502 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:05:51.188605 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260925 05:05:51.188733 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.179817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:56267 (local address 127.14.33.62:36881)
0925 05:05:51.179875 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:51.179877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:51.180046 (+ 169us) server_negotiation.cc:407] Connection header received
0925 05:05:51.180879 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:51.180880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:51.180889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:51.180943 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:51.181574 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.182053 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.183443 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.183644 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.183693 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.185828 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:51.185830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:51.185830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:51.185837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:51.187459 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.187830 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.187831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.187832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.187891 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.188102 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.188103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.188103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.188245 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:51.188333 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:51.188547 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 05:05:51.188633 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:05:51.189252 18361 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36881
I20260925 05:05:51.189355 18361 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:51.189536 18361 heartbeater.cc:511] Master 127.14.33.62:36881 requested a full tablet report, sending...
I20260925 05:05:51.189868 18036 ts_manager.cc:194] Registered new tserver with Master: 843a3057a8de40dcbcb96e382b5222c4 (127.14.33.2:35947)
I20260925 05:05:51.190223 18036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:56267
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:51.207288 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36881
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:51.287668 18366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:51.287889 18366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:51.287912 18366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:51.289211 18366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:51.289254 18366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:51.289287 18366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:51.290920 18366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:51.291177 18366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:51.291430 18366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:51.296538 18374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:51.296537 18372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:51.296650 18371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:51.296829 18366 server_base.cc:1034] running on GCE node
I20260925 05:05:51.297047 18366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:51.297335 18366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:51.298491 18366 hybrid_clock.cc:648] HybridClock initialized: now 1790312751298470 us; error 34 us; skew 500 ppm
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:51.301357 18366 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:51.301846 18366 webserver.cc:460] Webserver started at http://127.14.33.3:39865/ using document root <none> and password file <none>
I20260925 05:05:51.302073 18366 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:51.302119 18366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:51.302241 18366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:51.303373 18366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "059f918697ba4e0390ce2e9d1de80871"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "d1dc209244776de4c4ad63018c9ffdc5"
server_key_iv: "08fc8e23ea3d5da0dfef34ba3ea5a64e"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.303745 18366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "059f918697ba4e0390ce2e9d1de80871"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "d1dc209244776de4c4ad63018c9ffdc5"
server_key_iv: "08fc8e23ea3d5da0dfef34ba3ea5a64e"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.305348 18366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 05:05:51.306298 18381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.306523 18366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:51.306615 18366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "059f918697ba4e0390ce2e9d1de80871"
format_stamp: "Formatted at 2026-09-25 05:05:51 on dist-test-slave-4dmq"
server_key: "d1dc209244776de4c4ad63018c9ffdc5"
server_key_iv: "08fc8e23ea3d5da0dfef34ba3ea5a64e"
server_key_version: "encryptionkey@0"
I20260925 05:05:51.306689 18366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:51.327270 18366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:51.329283 18366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:51.329442 18366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:51.329710 18366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:51.330021 18366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:51.330057 18366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.330078 18366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:51.330123 18366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:51.336022 18366 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:34837
I20260925 05:05:51.336069 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:34837 every 8 connection(s)
I20260925 05:05:51.336370 18366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:51.341701 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18366
I20260925 05:05:51.341818 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:51.342011 14468 external_mini_cluster.cc:1428] Setting key fbf60ab86e5d47ceee87492ba6b5d7ef
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:51.345255 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.337042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:33167 (local address 127.14.33.62:36881)
0925 05:05:51.337137 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:51.337140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:51.337319 (+ 179us) server_negotiation.cc:407] Connection header received
0925 05:05:51.338200 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:51.338201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:51.338212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:51.338277 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:51.338815 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.339303 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.340231 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.340414 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.340465 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.342608 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:51.342610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:51.342611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:51.342618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:51.343845 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.344276 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.344278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.344278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.344340 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.344592 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.344592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.344593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.344718 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:51.344794 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:51.345053 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 05:05:51.345156 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:05:51.345428 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.336991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36881 (local address 127.14.33.3:33167)
0925 05:05:51.337199 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:51.337240 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:51.338114 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:51.338317 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:51.338320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:51.338558 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:51.338746 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:51.338750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.339380 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.339381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:51.340152 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:51.340154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.340484 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.340485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:51.340577 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.340993 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:51.341000 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:51.342521 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:51.343959 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:51.343965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:51.343967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:51.344183 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:51.344407 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:51.344408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:51.344409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:51.344517 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:51.344785 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:51.344791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:51.344925 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 05:05:51.345032 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:05:51.345204 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260925 05:05:51.345857 18495 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36881
I20260925 05:05:51.345953 18495 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:51.346122 18495 heartbeater.cc:511] Master 127.14.33.62:36881 requested a full tablet report, sending...
I20260925 05:05:51.346451 18036 ts_manager.cc:194] Registered new tserver with Master: 059f918697ba4e0390ce2e9d1de80871 (127.14.33.3:34837)
I20260925 05:05:51.346823 18036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:33167
I20260925 05:05:51.353590 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:51.405638 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.397317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.14.33.1:39195)
0925 05:05:51.397476 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:51.397482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:51.397501 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:05:51.397527 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:51.397529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:51.397542 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:51.397610 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:51.398355 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.399076 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.400260 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.400575 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.400627 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.403166 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:51.403175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:51.403175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:51.403182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:51.404564 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.404884 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.404887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.404887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.404944 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.405108 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.405109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.405109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.405307 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:51.405421 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:51.405490 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:05:51.405549 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
Sep 25 05:05:51 dist-test-slave-4dmq krb5kdc[17991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:51.413717 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:51.406991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.14.33.62:36881)
0925 05:05:51.407039 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:51.407042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:51.407216 (+ 174us) server_negotiation.cc:407] Connection header received
0925 05:05:51.407254 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:51.407254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:51.407268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:51.407344 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:51.407929 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.408528 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.409233 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:51.409412 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:51.409463 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:51.411538 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:51.411540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:51.411540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:51.411546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:51.412801 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.413094 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.413095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.413095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.413188 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:51.413321 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:51.413321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:51.413321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:51.413443 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:51.413513 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:51.413592 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 05:05:51.413646 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 05:05:51.413980 18036 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33220
I20260925 05:05:51.414351 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18098
I20260925 05:05:51.419108 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18232
I20260925 05:05:51.423530 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18366
I20260925 05:05:51.427871 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18004
2026-09-25T05:05:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1760 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:52 dist-test-slave-4dmq krb5kdc[18507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:52 dist-test-slave-4dmq krb5kdc[18507](info): set up 2 sockets
Sep 25 05:05:52 dist-test-slave-4dmq krb5kdc[18507](info): commencing operation
krb5kdc: starting...
W20260925 05:05:53.066291 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.619s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:53.173370 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36751
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:53.251231 18520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:53.251410 18520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:53.251430 18520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:53.252533 18520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:53.252571 18520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:53.252584 18520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:53.252620 18520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:53.252651 18520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:53.254060 18520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:36751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:53.254294 18520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:53.254504 18520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:53.259059 18526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:53.259059 18525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:53.259176 18520 server_base.cc:1034] running on GCE node
W20260925 05:05:53.259095 18528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:53.259552 18520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:53.259845 18520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:53.261016 18520 hybrid_clock.cc:648] HybridClock initialized: now 1790312753260994 us; error 32 us; skew 500 ppm
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:53.268563 18520 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:53.268899 18520 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:53.268980 18520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:53.269171 18520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:53.270221 18520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "4c4fb8414aa045b99f94b47a73914aeb"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "f32d27c54e9f5ed7ad8eecc2cb6c7c10"
server_key_iv: "b1f3fb6270278764ca28b4937479efeb"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.270579 18520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "4c4fb8414aa045b99f94b47a73914aeb"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "f32d27c54e9f5ed7ad8eecc2cb6c7c10"
server_key_iv: "b1f3fb6270278764ca28b4937479efeb"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.271845 18520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:05:53.272531 18534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.272698 18520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:53.272795 18520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "4c4fb8414aa045b99f94b47a73914aeb"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "f32d27c54e9f5ed7ad8eecc2cb6c7c10"
server_key_iv: "b1f3fb6270278764ca28b4937479efeb"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.272876 18520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:53.288069 18520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:53.289872 18520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:53.290026 18520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:53.294297 18520 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:36751
I20260925 05:05:53.294322 18586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:36751 every 8 connection(s)
I20260925 05:05:53.294584 18520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:53.295502 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:53.297264 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18520
I20260925 05:05:53.297351 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:53.297538 14468 external_mini_cluster.cc:1428] Setting key d9070def64b574fd87a4c6e8e146563a
I20260925 05:05:53.298048 18587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb: Bootstrap starting.
I20260925 05:05:53.298683 18587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:53.299018 18587 log.cc:824] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:53.299712 18587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb: No bootstrap required, opened a new log
I20260925 05:05:53.301474 18587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4fb8414aa045b99f94b47a73914aeb" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36751 } }
I20260925 05:05:53.301620 18587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:53.301662 18587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4fb8414aa045b99f94b47a73914aeb, State: Initialized, Role: FOLLOWER
I20260925 05:05:53.301792 18587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4fb8414aa045b99f94b47a73914aeb" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36751 } }
I20260925 05:05:53.301882 18587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:53.301920 18587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:53.301967 18587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:53.302460 18587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4fb8414aa045b99f94b47a73914aeb" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36751 } }
I20260925 05:05:53.302591 18587 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c4fb8414aa045b99f94b47a73914aeb; no voters:
I20260925 05:05:53.302740 18587 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:53.302814 18592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:53.302963 18592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [term 1 LEADER]: Becoming Leader. State: Replica: 4c4fb8414aa045b99f94b47a73914aeb, State: Running, Role: LEADER
I20260925 05:05:53.303053 18587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:53.303114 18592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4fb8414aa045b99f94b47a73914aeb" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36751 } }
I20260925 05:05:53.303565 18592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4fb8414aa045b99f94b47a73914aeb. Latest consensus state: current_term: 1 leader_uuid: "4c4fb8414aa045b99f94b47a73914aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4fb8414aa045b99f94b47a73914aeb" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36751 } } }
I20260925 05:05:53.303665 18594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c4fb8414aa045b99f94b47a73914aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4fb8414aa045b99f94b47a73914aeb" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36751 } } }
I20260925 05:05:53.303771 18594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:53.303757 18592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:53.304144 18601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:53.304857 18601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:53.307080 18601 catalog_manager.cc:1347] Generated new cluster ID: 2ae0ba5d1ccd464d8dfbc024607af09f
I20260925 05:05:53.307127 18601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:53.308535 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.298046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.14.33.62:36751)
0925 05:05:53.298206 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.298215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.298244 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:05:53.298892 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.298901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.298972 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.299132 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:53.299877 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.300713 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.301575 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.302143 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.302305 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.305152 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:53.305156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:53.305156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:53.305167 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:53.306932 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.307339 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.307342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.307342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.307448 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.307656 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.307657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.307657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.307955 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:53.308114 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.308193 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.308338 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260925 05:05:53.318791 18601 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:53.319288 18601 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:53.323884 18601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb: Generated new TSK 0
I20260925 05:05:53.324043 18601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:53.351362 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36751
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:53.429096 18613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:53.429343 18613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:53.429368 18613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:53.430469 18613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:53.430507 18613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:53.430553 18613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:53.431867 18613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:53.432066 18613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:53.432267 18613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:53.436664 18618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:53.436695 18613 server_base.cc:1034] running on GCE node
W20260925 05:05:53.436657 18619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:53.436664 18621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:53.437086 18613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:53.437364 18613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:53.438522 18613 hybrid_clock.cc:648] HybridClock initialized: now 1790312753438511 us; error 24 us; skew 500 ppm
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:53.440887 18613 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:53.441206 18613 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:53.441253 18613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:53.441334 18613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:53.442204 18613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "209aa2215ec94030a11c954b6cc3bebc"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "97fd585b7bfc837bb6efe47bec027858"
server_key_iv: "3dde4bd5425c5dbb799b2131dba2079c"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.442478 18613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "209aa2215ec94030a11c954b6cc3bebc"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "97fd585b7bfc837bb6efe47bec027858"
server_key_iv: "3dde4bd5425c5dbb799b2131dba2079c"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.443622 18613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:53.444278 18627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.444418 18613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:53.444494 18613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "209aa2215ec94030a11c954b6cc3bebc"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "97fd585b7bfc837bb6efe47bec027858"
server_key_iv: "3dde4bd5425c5dbb799b2131dba2079c"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.444541 18613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:53.451349 18613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:53.453526 18613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:53.453625 18613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:53.453784 18613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:53.454005 18613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:53.454033 18613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.454052 18613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:53.454064 18613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.459244 18613 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39747
I20260925 05:05:53.459313 18740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39747 every 8 connection(s)
I20260925 05:05:53.459553 18613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:53.465523 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18613
I20260925 05:05:53.465662 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:53.465844 14468 external_mini_cluster.cc:1428] Setting key bdd7727151d6a9519cc5ce51c6285272
I20260925 05:05:53.469441 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.460179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:48553 (local address 127.14.33.62:36751)
0925 05:05:53.460236 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.460238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.460480 (+ 242us) server_negotiation.cc:407] Connection header received
0925 05:05:53.460942 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.460943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.460955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.461009 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:53.461899 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.462391 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.463338 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.463542 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.463624 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.466251 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:53.466253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:53.466253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:53.466260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:53.467909 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.468317 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.468319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.468319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.468374 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.468658 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.468659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.468660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.468817 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:53.468893 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.469230 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.469325 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:05:53.469472 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.460176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36751 (local address 127.14.33.1:48553)
0925 05:05:53.460346 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.460400 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.460833 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.461066 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.461069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.461367 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:53.461796 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.461800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.462480 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.462482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.463236 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.463238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.463633 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.463634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.463773 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.464296 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:53.464309 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:53.466128 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:53.468004 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:53.468012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:53.468014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:53.468227 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:53.468467 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:53.468468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:53.468469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:53.468586 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:53.468920 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:53.468927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:53.469055 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.469202 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.469328 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260925 05:05:53.469918 18741 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36751
I20260925 05:05:53.470008 18741 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:53.470149 18741 heartbeater.cc:511] Master 127.14.33.62:36751 requested a full tablet report, sending...
I20260925 05:05:53.470578 18551 ts_manager.cc:194] Registered new tserver with Master: 209aa2215ec94030a11c954b6cc3bebc (127.14.33.1:39747)
I20260925 05:05:53.471249 18551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:48553
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:53.490036 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36751
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:53.578967 18746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:53.579191 18746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:53.579241 18746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:53.580870 18746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:53.580933 18746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:53.580976 18746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:53.583293 18746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:53.583527 18746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:53.583714 18746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:53.588507 18751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:53.588500 18752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:53.588491 18754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:53.588745 18746 server_base.cc:1034] running on GCE node
I20260925 05:05:53.588973 18746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:53.589249 18746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:53.590401 18746 hybrid_clock.cc:648] HybridClock initialized: now 1790312753590382 us; error 37 us; skew 500 ppm
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:53.593298 18746 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:53.593796 18746 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:53.593845 18746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:53.594000 18746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:53.595010 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "fc2d7132cfb143c49dbe654787d12f70"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "d5cbb24f92ec2cbfb1faedba6a5950c8"
server_key_iv: "75835cd377f7732ae4f17102866b9c5f"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.595355 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "fc2d7132cfb143c49dbe654787d12f70"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "d5cbb24f92ec2cbfb1faedba6a5950c8"
server_key_iv: "75835cd377f7732ae4f17102866b9c5f"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.596654 18746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:53.597414 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.597604 18746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:53.597704 18746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "fc2d7132cfb143c49dbe654787d12f70"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "d5cbb24f92ec2cbfb1faedba6a5950c8"
server_key_iv: "75835cd377f7732ae4f17102866b9c5f"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.597788 18746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:53.610113 18746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:53.611846 18746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:53.612018 18746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:53.612223 18746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:53.612525 18746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:53.612581 18746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.612620 18746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:53.612649 18746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.618283 18746 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:43099
I20260925 05:05:53.618330 18873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:43099 every 8 connection(s)
I20260925 05:05:53.618638 18746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:53.624776 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18746
I20260925 05:05:53.624889 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:53.625118 14468 external_mini_cluster.cc:1428] Setting key ffe19865b8c606959bd0c79040737ae2
I20260925 05:05:53.627822 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.619386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36751 (local address 127.14.33.2:60089)
0925 05:05:53.619558 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.619591 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.620022 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.620230 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.620233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.620462 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:53.620919 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.620924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.621619 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.621619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.622363 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.622365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.622703 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.622703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.622794 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.623209 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:53.623216 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:53.624525 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:53.626351 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:53.626358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:53.626360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:53.626663 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:53.626864 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:53.626864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:53.626865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:53.627011 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:53.627284 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:53.627290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:53.627457 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.627560 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.627668 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260925 05:05:53.627815 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.619396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:60089 (local address 127.14.33.62:36751)
0925 05:05:53.619462 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.619464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.619675 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:05:53.620102 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.620104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.620114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.620177 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:53.620993 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.621540 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.622442 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.622613 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.622679 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.624636 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:53.624639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:53.624639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:53.624656 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:53.626272 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.626752 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.626753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.626753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.626802 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.627070 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.627070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.627071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.627186 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:53.627275 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.627607 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.627698 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:05:53.628304 18874 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36751
I20260925 05:05:53.628413 18874 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:53.628590 18874 heartbeater.cc:511] Master 127.14.33.62:36751 requested a full tablet report, sending...
I20260925 05:05:53.628957 18551 ts_manager.cc:194] Registered new tserver with Master: fc2d7132cfb143c49dbe654787d12f70 (127.14.33.2:43099)
I20260925 05:05:53.629400 18551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:60089
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:53.648561 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36751
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:53.730530 18879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:53.730747 18879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:53.730786 18879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:53.731874 18879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:53.731946 18879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:53.731993 18879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:53.733466 18879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:53.733713 18879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:53.733942 18879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:53.739223 18884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:53.739223 18887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:53.739311 18885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:53.739359 18879 server_base.cc:1034] running on GCE node
I20260925 05:05:53.739696 18879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:53.739923 18879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:53.741084 18879 hybrid_clock.cc:648] HybridClock initialized: now 1790312753741075 us; error 38 us; skew 500 ppm
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:53.743532 18879 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:53.743854 18879 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:53.743933 18879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:53.744108 18879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:53.745110 18879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "a408933eeaf7468fbc73d6a2c1f8622c"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "229523c41ba4711cc0151b8dd3505559"
server_key_iv: "0b6546aa7220fe156ec692ec4cb3f651"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.745522 18879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "a408933eeaf7468fbc73d6a2c1f8622c"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "229523c41ba4711cc0151b8dd3505559"
server_key_iv: "0b6546aa7220fe156ec692ec4cb3f651"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.746824 18879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:53.747548 18893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.747718 18879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:53.747826 18879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "a408933eeaf7468fbc73d6a2c1f8622c"
format_stamp: "Formatted at 2026-09-25 05:05:53 on dist-test-slave-4dmq"
server_key: "229523c41ba4711cc0151b8dd3505559"
server_key_iv: "0b6546aa7220fe156ec692ec4cb3f651"
server_key_version: "encryptionkey@0"
I20260925 05:05:53.747898 18879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:53.767915 18879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:53.770068 18879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:53.770215 18879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:53.770426 18879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:53.770699 18879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:53.770748 18879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.770797 18879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:53.770824 18879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:53.776715 18879 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:39863
I20260925 05:05:53.776786 19006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:39863 every 8 connection(s)
I20260925 05:05:53.777050 18879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:05:53 dist-test-slave-4dmq krb5kdc[18507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:53.783582 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 18879
I20260925 05:05:53.783735 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:53.783958 14468 external_mini_cluster.cc:1428] Setting key 08bf09ee318e5b36ea3f31a7f97a7f73
I20260925 05:05:53.785738 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.777701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:60519 (local address 127.14.33.62:36751)
0925 05:05:53.777766 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.777768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.777944 (+ 176us) server_negotiation.cc:407] Connection header received
0925 05:05:53.778375 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.778376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.778385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.778440 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:53.779250 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.779731 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.780632 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.780810 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.780858 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.782796 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:53.782798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:53.782798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:53.782804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:53.784387 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.784763 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.784764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.784764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.784819 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.785069 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.785070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.785070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.785240 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:53.785312 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.785576 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.785651 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:05:53.785913 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.777685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36751 (local address 127.14.33.3:60519)
0925 05:05:53.777827 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.777864 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.778283 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.778490 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.778493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.778733 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:53.779178 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.779183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.779805 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.779806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.780561 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.780563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.780877 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.780878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.780967 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.781409 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:53.781417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:53.782719 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:53.784464 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:53.784470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:53.784472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:53.784678 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:53.784893 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:53.784894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:53.784895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:53.784995 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:53.785316 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:53.785323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:53.785446 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.785543 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.785684 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260925 05:05:53.786309 19007 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36751
I20260925 05:05:53.786378 19007 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:53.786535 19007 heartbeater.cc:511] Master 127.14.33.62:36751 requested a full tablet report, sending...
I20260925 05:05:53.786867 18551 ts_manager.cc:194] Registered new tserver with Master: a408933eeaf7468fbc73d6a2c1f8622c (127.14.33.3:39863)
I20260925 05:05:53.787238 18551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:60519
I20260925 05:05:53.795514 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:53.802695 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.796814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.14.33.62:36751)
0925 05:05:53.796878 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.796880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.796962 (+ 82us) server_negotiation.cc:407] Connection header received
0925 05:05:53.796996 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.796997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.797005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.797054 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:53.797646 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.798173 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.798868 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.799051 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.799113 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.800240 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:53.800241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:53.800241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:53.800247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:53.801597 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.801921 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.801922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.801922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.801977 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:53.802158 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:53.802159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:53.802159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:53.802292 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:53.802372 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.802516 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.802579 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:05:53.804378 18551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:53.806077 18551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:53.812220 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.808264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:39863 (local address 127.0.0.1:33064)
0925 05:05:53.808379 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.808388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.808425 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.808738 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.808739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.808742 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:53.808794 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.808796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.809779 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.809781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.811054 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.811057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.811928 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.812006 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.812019 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.812089 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.812138 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":37,"thread_start_us":51,"threads_started":1}
I20260925 05:05:53.812237 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.808443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.14.33.3:39863)
0925 05:05:53.808574 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.808579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.808587 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:53.808625 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.808628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.808642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.808726 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:53.808902 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.809684 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811130 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.811836 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811915 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.811976 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.812111 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.812158 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260925 05:05:53.812453 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.808038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39747 (local address 127.0.0.1:45508)
0925 05:05:53.808349 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.808361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.808423 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.808685 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.808686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.808687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:53.808714 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.808715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.810069 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.810070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.811136 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.811139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.812025 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.812026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.812119 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.812122 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.812171 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.812209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260925 05:05:53.812803 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.808147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.14.33.1:39747)
0925 05:05:53.808330 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.808335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.808483 (+ 148us) server_negotiation.cc:407] Connection header received
0925 05:05:53.808524 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.808526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.808540 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.808604 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:53.808808 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.809979 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811216 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.811885 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.811993 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.812674 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.812725 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20260925 05:05:53.813036 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.808039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:43099 (local address 127.0.0.1:47552)
0925 05:05:53.808185 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.808220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.808277 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.808648 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.808651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.808653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:53.808711 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.808713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.811218 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.811853 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.811854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.812799 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.812800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.812869 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.812872 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.812927 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.812963 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"spinlock_wait_cycles":896,"thread_start_us":54,"threads_started":1}
I20260925 05:05:53.812922 18941 tablet_service.cc:1467] Processing CreateTablet for tablet 3530afe394a44d44940d411e2ec5bf12 (DEFAULT_TABLE table=test-table [id=182423b7d8a64668b605659a2a35612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:53.813068 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.808147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.14.33.2:43099)
0925 05:05:53.808404 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.808407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.808418 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:05:53.808457 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.808460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.808477 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.808553 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:53.808858 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.809963 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.811965 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.812705 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.812788 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.812843 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.812942 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.812978 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":78,"threads_started":1}
I20260925 05:05:53.813297 18941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3530afe394a44d44940d411e2ec5bf12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:53.813608 18808 tablet_service.cc:1467] Processing CreateTablet for tablet 3530afe394a44d44940d411e2ec5bf12 (DEFAULT_TABLE table=test-table [id=182423b7d8a64668b605659a2a35612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:53.813591 18675 tablet_service.cc:1467] Processing CreateTablet for tablet 3530afe394a44d44940d411e2ec5bf12 (DEFAULT_TABLE table=test-table [id=182423b7d8a64668b605659a2a35612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:53.813831 18808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3530afe394a44d44940d411e2ec5bf12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:53.813884 18675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3530afe394a44d44940d411e2ec5bf12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:53.816015 19021 tablet_bootstrap.cc:492] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Bootstrap starting.
I20260925 05:05:53.816455 19022 tablet_bootstrap.cc:492] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Bootstrap starting.
I20260925 05:05:53.816692 19021 tablet_bootstrap.cc:654] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:53.816953 19023 tablet_bootstrap.cc:492] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: Bootstrap starting.
I20260925 05:05:53.817091 19022 tablet_bootstrap.cc:654] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:53.817342 19021 log.cc:824] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:53.817616 19023 tablet_bootstrap.cc:654] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:53.817704 19022 log.cc:824] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:53.818228 19023 log.cc:824] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:53.818312 19021 tablet_bootstrap.cc:492] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: No bootstrap required, opened a new log
I20260925 05:05:53.818365 19021 ts_tablet_manager.cc:1397] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:53.818576 19022 tablet_bootstrap.cc:492] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: No bootstrap required, opened a new log
I20260925 05:05:53.818657 19022 ts_tablet_manager.cc:1397] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:53.818943 19023 tablet_bootstrap.cc:492] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: No bootstrap required, opened a new log
I20260925 05:05:53.819010 19023 ts_tablet_manager.cc:1397] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:53.820209 19022 raft_consensus.cc:348] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.820363 19022 raft_consensus.cc:374] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:53.820406 19022 raft_consensus.cc:729] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a408933eeaf7468fbc73d6a2c1f8622c, State: Initialized, Role: FOLLOWER
I20260925 05:05:53.820369 19021 raft_consensus.cc:348] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.820526 19021 raft_consensus.cc:374] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:53.820566 19021 raft_consensus.cc:729] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2d7132cfb143c49dbe654787d12f70, State: Initialized, Role: FOLLOWER
I20260925 05:05:53.820544 19022 consensus_queue.cc:260] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.820680 19021 consensus_queue.cc:260] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.820811 19022 ts_tablet_manager.cc:1428] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:05:53.820856 19007 heartbeater.cc:503] Master 127.14.33.62:36751 was elected leader, sending a full tablet report...
I20260925 05:05:53.820952 19021 ts_tablet_manager.cc:1428] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:53.820945 19023 raft_consensus.cc:348] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.821063 18874 heartbeater.cc:503] Master 127.14.33.62:36751 was elected leader, sending a full tablet report...
I20260925 05:05:53.821082 19023 raft_consensus.cc:374] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:53.821118 19023 raft_consensus.cc:729] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209aa2215ec94030a11c954b6cc3bebc, State: Initialized, Role: FOLLOWER
I20260925 05:05:53.821266 19023 consensus_queue.cc:260] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.821528 19023 ts_tablet_manager.cc:1428] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 05:05:53.821563 18741 heartbeater.cc:503] Master 127.14.33.62:36751 was elected leader, sending a full tablet report...
W20260925 05:05:53.869156 18875 tablet.cc:2369] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:53.931047 19029 raft_consensus.cc:482] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:53.931171 19029 raft_consensus.cc:504] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.931461 19029 leader_election.cc:288] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc2d7132cfb143c49dbe654787d12f70 (127.14.33.2:43099), a408933eeaf7468fbc73d6a2c1f8622c (127.14.33.3:39863)
I20260925 05:05:53.934250 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.931745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:59459 (local address 127.14.33.2:43099)
0925 05:05:53.931802 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.931805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.931815 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:05:53.931851 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.931852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.931865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.931921 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:53.932094 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.932557 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.933548 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.933913 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.933993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.934003 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.934157 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.934189 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:05:53.934250 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.931654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:43099 (local address 127.14.33.1:59459)
0925 05:05:53.931720 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.931733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.931799 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.931938 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.931939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.931940 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:53.932009 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.932011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.932640 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.932641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.933449 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.933451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.933992 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.933993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.934068 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.934078 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.934139 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.934177 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:05:53.934536 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.931863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:56155 (local address 127.14.33.3:39863)
0925 05:05:53.931927 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.931929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.932047 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:05:53.932106 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.932107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.932118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.932171 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:05:53.932353 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.932808 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.933790 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.934151 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.934239 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.934247 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.934429 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.934475 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:05:53.934578 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530afe394a44d44940d411e2ec5bf12" candidate_uuid: "209aa2215ec94030a11c954b6cc3bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2d7132cfb143c49dbe654787d12f70" is_pre_election: true
I20260925 05:05:53.934741 18828 raft_consensus.cc:2454] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209aa2215ec94030a11c954b6cc3bebc in term 0.
I20260925 05:05:53.934755 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.931765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:39863 (local address 127.14.33.1:56155)
0925 05:05:53.931978 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.931987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:53.932051 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:53.932185 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:53.932186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:53.932188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:05:53.932267 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.932268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.932893 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.932894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.933700 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:53.933703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.934241 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.934242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:53.934332 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.934337 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:05:53.934633 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 05:05:53.934676 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
I20260925 05:05:53.934988 18628 leader_election.cc:302] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209aa2215ec94030a11c954b6cc3bebc, fc2d7132cfb143c49dbe654787d12f70; no voters:
I20260925 05:05:53.935106 18961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530afe394a44d44940d411e2ec5bf12" candidate_uuid: "209aa2215ec94030a11c954b6cc3bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" is_pre_election: true
I20260925 05:05:53.935146 19029 raft_consensus.cc:2781] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:53.935199 19029 raft_consensus.cc:482] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:53.935222 19029 raft_consensus.cc:3037] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:53.935253 18961 raft_consensus.cc:2454] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209aa2215ec94030a11c954b6cc3bebc in term 0.
I20260925 05:05:53.936012 19029 raft_consensus.cc:504] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.936141 19029 leader_election.cc:288] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [CANDIDATE]: Term 1 election: Requested vote from peers fc2d7132cfb143c49dbe654787d12f70 (127.14.33.2:43099), a408933eeaf7468fbc73d6a2c1f8622c (127.14.33.3:39863)
I20260925 05:05:53.936314 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530afe394a44d44940d411e2ec5bf12" candidate_uuid: "209aa2215ec94030a11c954b6cc3bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2d7132cfb143c49dbe654787d12f70"
I20260925 05:05:53.936332 18961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3530afe394a44d44940d411e2ec5bf12" candidate_uuid: "209aa2215ec94030a11c954b6cc3bebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a408933eeaf7468fbc73d6a2c1f8622c"
I20260925 05:05:53.936398 18961 raft_consensus.cc:3037] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:53.936399 18828 raft_consensus.cc:3037] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:53.937282 18828 raft_consensus.cc:2454] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209aa2215ec94030a11c954b6cc3bebc in term 1.
I20260925 05:05:53.937282 18961 raft_consensus.cc:2454] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209aa2215ec94030a11c954b6cc3bebc in term 1.
I20260925 05:05:53.937462 18628 leader_election.cc:302] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209aa2215ec94030a11c954b6cc3bebc, a408933eeaf7468fbc73d6a2c1f8622c; no voters:
I20260925 05:05:53.937551 19029 raft_consensus.cc:2781] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:53.937691 19029 raft_consensus.cc:686] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [term 1 LEADER]: Becoming Leader. State: Replica: 209aa2215ec94030a11c954b6cc3bebc, State: Running, Role: LEADER
I20260925 05:05:53.937788 19029 consensus_queue.cc:237] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } }
I20260925 05:05:53.938591 18550 catalog_manager.cc:5495] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc reported cstate change: term changed from 0 to 1, leader changed from <none> to 209aa2215ec94030a11c954b6cc3bebc (127.14.33.1). New cstate: current_term: 1 leader_uuid: "209aa2215ec94030a11c954b6cc3bebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "209aa2215ec94030a11c954b6cc3bebc" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39747 } health_report { overall_health: HEALTHY } } }
I20260925 05:05:53.958323 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.955916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.14.33.1:39747)
0925 05:05:53.955969 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:53.955971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:53.955979 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:53.956049 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:53.956050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:53.956067 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:53.956136 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:05:53.956280 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.956728 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.957543 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:53.957742 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:53.957823 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:53.957991 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:05:53.958096 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:05:53.958188 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:53.958207 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 05:05:53.958226 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:05:53.959992 18828 raft_consensus.cc:1261] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 1 FOLLOWER]: Refusing update from remote peer 209aa2215ec94030a11c954b6cc3bebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:53.959992 18961 raft_consensus.cc:1261] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 1 FOLLOWER]: Refusing update from remote peer 209aa2215ec94030a11c954b6cc3bebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 05:05:53.960273 18742 tablet.cc:2369] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:53.960369 19031 consensus_queue.cc:1035] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2d7132cfb143c49dbe654787d12f70" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:53.960507 19029 consensus_queue.cc:1035] T 3530afe394a44d44940d411e2ec5bf12 P 209aa2215ec94030a11c954b6cc3bebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a408933eeaf7468fbc73d6a2c1f8622c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:53.962733 19034 mvcc.cc:205] Tried to move back new op lower bound from 7333121040217612288 to 7333121040129748992. Current Snapshot: MvccSnapshot[applied={T|T < 7333121040217612288}]
I20260925 05:05:53.962910 19035 mvcc.cc:205] Tried to move back new op lower bound from 7333121040217612288 to 7333121040129748992. Current Snapshot: MvccSnapshot[applied={T|T < 7333121040217612288}]
I20260925 05:05:53.963655 19036 mvcc.cc:205] Tried to move back new op lower bound from 7333121040217612288 to 7333121040129748992. Current Snapshot: MvccSnapshot[applied={T|T < 7333121040217612288 or (T in {7333121040217612288})}]
I20260925 05:05:53.965065 18550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:53.965183 18550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:53.965950 18550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c4fb8414aa045b99f94b47a73914aeb: Sending DeleteTablet for 3 replicas of tablet 3530afe394a44d44940d411e2ec5bf12
I20260925 05:05:53.966394 18675 tablet_service.cc:1514] Processing DeleteTablet for tablet 3530afe394a44d44940d411e2ec5bf12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:53 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:45508
I20260925 05:05:53.966398 18941 tablet_service.cc:1514] Processing DeleteTablet for tablet 3530afe394a44d44940d411e2ec5bf12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:53 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:33064
I20260925 05:05:53.966624 18808 tablet_service.cc:1514] Processing DeleteTablet for tablet 3530afe394a44d44940d411e2ec5bf12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:53 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:47552
I20260925 05:05:53.966722 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18613
I20260925 05:05:53.968065 19043 tablet_replica.cc:330] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: stopping tablet replica
I20260925 05:05:53.968083 19045 tablet_replica.cc:330] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: stopping tablet replica
I20260925 05:05:53.968186 19045 raft_consensus.cc:2229] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:53.968190 19043 raft_consensus.cc:2229] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:53.968284 19045 raft_consensus.cc:2258] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:53.968290 19043 raft_consensus.cc:2258] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:53.968714 19043 ts_tablet_manager.cc:1905] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:53.968895 19045 ts_tablet_manager.cc:1905] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:53.970466 19043 ts_tablet_manager.cc:1918] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:53.970551 19043 log.cc:1196] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/3530afe394a44d44940d411e2ec5bf12
I20260925 05:05:53.970788 19045 ts_tablet_manager.cc:1918] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:53.970885 19043 ts_tablet_manager.cc:1939] T 3530afe394a44d44940d411e2ec5bf12 P a408933eeaf7468fbc73d6a2c1f8622c: Deleting consensus metadata
I20260925 05:05:53.970919 19045 log.cc:1196] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/3530afe394a44d44940d411e2ec5bf12
I20260925 05:05:53.971380 18535 catalog_manager.cc:4841] TS a408933eeaf7468fbc73d6a2c1f8622c (127.14.33.3:39863): tablet 3530afe394a44d44940d411e2ec5bf12 (table test-table [id=182423b7d8a64668b605659a2a35612a]) successfully deleted
I20260925 05:05:53.971411 19045 ts_tablet_manager.cc:1939] T 3530afe394a44d44940d411e2ec5bf12 P fc2d7132cfb143c49dbe654787d12f70: Deleting consensus metadata
I20260925 05:05:53.971845 18535 catalog_manager.cc:4841] TS fc2d7132cfb143c49dbe654787d12f70 (127.14.33.2:43099): tablet 3530afe394a44d44940d411e2ec5bf12 (table test-table [id=182423b7d8a64668b605659a2a35612a]) successfully deleted
W20260925 05:05:53.972182 18538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:39747 (error 108)
I20260925 05:05:53.972366 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18746
I20260925 05:05:53.973206 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:53.973031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39747 (local address 127.0.0.1:45528)
0925 05:05:53.973092 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:53.973159 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 05:05:53.974124 18538 catalog_manager.cc:4584] TS 209aa2215ec94030a11c954b6cc3bebc (127.14.33.1:39747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3530afe394a44d44940d411e2ec5bf12: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39747: connect: Connection refused (error 111)
I20260925 05:05:53.977574 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18879
I20260925 05:05:53.982046 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 18520
2026-09-25T05:05:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2554 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T05:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:54Z Disabled control of system clock
I20260925 05:05:54.009271 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38337
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:05:54.088349 19050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:54.088551 19050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:54.088572 19050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:54.089747 19050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:54.089787 19050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:54.089800 19050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:54.089812 19050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:54.089872 19050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:54.091075 19050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:54.091331 19050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:54.091600 19050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:54.096457 19058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.096553 19050 server_base.cc:1034] running on GCE node
W20260925 05:05:54.096453 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.096462 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.096971 19050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:54.097383 19050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:54.098548 19050 hybrid_clock.cc:648] HybridClock initialized: now 1790312754098527 us; error 32 us; skew 500 ppm
I20260925 05:05:54.099740 19050 webserver.cc:460] Webserver started at http://127.14.33.62:36149/ using document root <none> and password file <none>
I20260925 05:05:54.099982 19050 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:54.100059 19050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:54.100209 19050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:54.101271 19050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "bf24cb3a43df4175a527045d768a12a8"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "a4bdcfc03ca5b475211de3d191b40358"
server_key_iv: "3315e5d725479772f5f8a2b38f6dd348"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.101612 19050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "bf24cb3a43df4175a527045d768a12a8"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "a4bdcfc03ca5b475211de3d191b40358"
server_key_iv: "3315e5d725479772f5f8a2b38f6dd348"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.102775 19050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:54.103456 19064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.103626 19050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:54.103749 19050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "bf24cb3a43df4175a527045d768a12a8"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "a4bdcfc03ca5b475211de3d191b40358"
server_key_iv: "3315e5d725479772f5f8a2b38f6dd348"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.103843 19050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:54.107128 19050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:54.107384 19050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:54.107527 19050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:54.111346 19050 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38337
I20260925 05:05:54.111374 19116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38337 every 8 connection(s)
I20260925 05:05:54.111680 19050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:54.112279 19117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:54.113234 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19050
I20260925 05:05:54.113344 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:54.113516 14468 external_mini_cluster.cc:1428] Setting key 8e97e5ea168f9e5f0b37c9fbbb9e2972
I20260925 05:05:54.115832 19117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8: Bootstrap starting.
I20260925 05:05:54.116019 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.113990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.14.33.62:38337)
0925 05:05:54.114182 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.114189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.114213 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:05:54.114596 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.114601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.114637 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.114758 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.115535 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.115538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.115538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.115542 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.115579 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.115589 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.115643 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.115736 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.115802 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260925 05:05:54.116580 19117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:54.116846 19117 log.cc:824] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:54.117571 19117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8: No bootstrap required, opened a new log
I20260925 05:05:54.118871 19117 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf24cb3a43df4175a527045d768a12a8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38337 } }
I20260925 05:05:54.119017 19117 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:54.119063 19117 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf24cb3a43df4175a527045d768a12a8, State: Initialized, Role: FOLLOWER
I20260925 05:05:54.119181 19117 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf24cb3a43df4175a527045d768a12a8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38337 } }
I20260925 05:05:54.119271 19117 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:54.119314 19117 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:54.119362 19117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:54.120196 19117 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf24cb3a43df4175a527045d768a12a8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38337 } }
I20260925 05:05:54.120297 19117 leader_election.cc:302] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf24cb3a43df4175a527045d768a12a8; no voters:
I20260925 05:05:54.120443 19117 leader_election.cc:288] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:54.120473 19122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:54.120605 19122 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [term 1 LEADER]: Becoming Leader. State: Replica: bf24cb3a43df4175a527045d768a12a8, State: Running, Role: LEADER
I20260925 05:05:54.120700 19117 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:54.120699 19122 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf24cb3a43df4175a527045d768a12a8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38337 } }
I20260925 05:05:54.121161 19124 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf24cb3a43df4175a527045d768a12a8. Latest consensus state: current_term: 1 leader_uuid: "bf24cb3a43df4175a527045d768a12a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf24cb3a43df4175a527045d768a12a8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38337 } } }
I20260925 05:05:54.121157 19123 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf24cb3a43df4175a527045d768a12a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf24cb3a43df4175a527045d768a12a8" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38337 } } }
I20260925 05:05:54.121390 19124 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:54.121407 19123 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:54.121728 19131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:54.122224 19131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:54.123648 19131 catalog_manager.cc:1347] Generated new cluster ID: e0e960bcadd5404e846422b8bb43846a
I20260925 05:05:54.123705 19131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:54.154949 19131 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:54.155128 19131 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:54.170820 19131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8: Generated new TSK 0
I20260925 05:05:54.171010 19131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:54.176025 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38337
--builtin_ntp_servers=127.14.33.20:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:05:54.258777 19141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:54.258965 19141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:54.258986 19141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:54.260038 19141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:54.260071 19141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:54.260097 19141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:54.261394 19141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:54.261561 19141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:54.261778 19141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:54.266458 19149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.266479 19146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.266417 19147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.266645 19141 server_base.cc:1034] running on GCE node
I20260925 05:05:54.266847 19141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:54.267036 19141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:54.268203 19141 hybrid_clock.cc:648] HybridClock initialized: now 1790312754268187 us; error 31 us; skew 500 ppm
I20260925 05:05:54.269440 19141 webserver.cc:460] Webserver started at http://127.14.33.1:45223/ using document root <none> and password file <none>
I20260925 05:05:54.269657 19141 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:54.269701 19141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:54.269834 19141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:54.270849 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "0584cf715144449cbee16830dccf034e"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "a31b5fd0abef5a9e1b15c4526f23a841"
server_key_iv: "38cf1ee5abea1a9bae3528deb2cb844c"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.271188 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "0584cf715144449cbee16830dccf034e"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "a31b5fd0abef5a9e1b15c4526f23a841"
server_key_iv: "38cf1ee5abea1a9bae3528deb2cb844c"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.272398 19141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:54.273059 19155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.273263 19141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:54.273366 19141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "0584cf715144449cbee16830dccf034e"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "a31b5fd0abef5a9e1b15c4526f23a841"
server_key_iv: "38cf1ee5abea1a9bae3528deb2cb844c"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.273442 19141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:54.276503 19141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:54.276748 19141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:54.276885 19141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:54.277081 19141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:54.277397 19141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:54.277462 19141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.277498 19141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:54.277524 19141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.282984 19141 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:44769
I20260925 05:05:54.283047 19268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:44769 every 8 connection(s)
I20260925 05:05:54.283358 19141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:54.286176 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.284055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:52881 (local address 127.14.33.62:38337)
0925 05:05:54.284131 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.284134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.284331 (+ 197us) server_negotiation.cc:407] Connection header received
0925 05:05:54.284747 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.284748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.284754 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.284807 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.285962 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.285964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.285964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.285968 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.285984 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.285988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.286043 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.286101 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.286115 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:05:54.286351 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.284016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38337 (local address 127.14.33.1:52881)
0925 05:05:54.284209 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.284240 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.284668 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.284856 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.284859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.284866 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.285825 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.285826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.285852 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.285878 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.286043 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.286045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.286047 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.286097 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.286140 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260925 05:05:54.286918 19269 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38337
I20260925 05:05:54.287011 19269 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:54.287154 19269 heartbeater.cc:511] Master 127.14.33.62:38337 requested a full tablet report, sending...
I20260925 05:05:54.287659 19081 ts_manager.cc:194] Registered new tserver with Master: 0584cf715144449cbee16830dccf034e (127.14.33.1:44769)
I20260925 05:05:54.290388 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19141
I20260925 05:05:54.290467 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:54.290656 14468 external_mini_cluster.cc:1428] Setting key 893175fa81c570b4313fee784509826b
I20260925 05:05:54.291572 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38337
--builtin_ntp_servers=127.14.33.20:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:05:54.372220 19272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:54.372416 19272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:54.372438 19272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:54.373612 19272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:54.373651 19272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:54.373677 19272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:54.374915 19272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:54.375088 19272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:54.375320 19272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:54.380355 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.380373 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.380362 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.380357 19272 server_base.cc:1034] running on GCE node
I20260925 05:05:54.380748 19272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:54.381009 19272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:54.382148 19272 hybrid_clock.cc:648] HybridClock initialized: now 1790312754382127 us; error 36 us; skew 500 ppm
I20260925 05:05:54.383395 19272 webserver.cc:460] Webserver started at http://127.14.33.2:34891/ using document root <none> and password file <none>
I20260925 05:05:54.383625 19272 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:54.383701 19272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:54.383836 19272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:54.384877 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "1703837391ff42538d6bd508830992a1"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "04ddf966bd7c5c34ecebc27965580f9a"
server_key_iv: "479e991a9896a90a8f3bf956d43ab74d"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.385250 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "1703837391ff42538d6bd508830992a1"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "04ddf966bd7c5c34ecebc27965580f9a"
server_key_iv: "479e991a9896a90a8f3bf956d43ab74d"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.386456 19272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:05:54.387174 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.387382 19272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:54.387475 19272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "1703837391ff42538d6bd508830992a1"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "04ddf966bd7c5c34ecebc27965580f9a"
server_key_iv: "479e991a9896a90a8f3bf956d43ab74d"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.387560 19272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:54.391016 19272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:54.391276 19272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:54.391410 19272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:54.391613 19272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:54.391916 19272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:54.391973 19272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.392009 19272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:54.392037 19272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.397562 19272 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:44475
I20260925 05:05:54.397650 19399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:44475 every 8 connection(s)
I20260925 05:05:54.397928 19272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:54.401048 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.398612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:60057 (local address 127.14.33.62:38337)
0925 05:05:54.398674 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.398676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.398914 (+ 238us) server_negotiation.cc:407] Connection header received
0925 05:05:54.399478 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.399479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.399486 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.399533 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.400780 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.400781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.400782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.400786 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.400801 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.400805 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.400850 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.400962 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.400977 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:05:54.401250 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.398602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38337 (local address 127.14.33.2:60057)
0925 05:05:54.398807 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.398845 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.399315 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.399606 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.399610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.399618 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.400637 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.400637 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.400661 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.400686 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.400858 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.400863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.400865 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.400972 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.401029 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260925 05:05:54.401732 19400 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38337
I20260925 05:05:54.401820 19400 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:54.401917 19400 heartbeater.cc:511] Master 127.14.33.62:38337 requested a full tablet report, sending...
I20260925 05:05:54.402240 19081 ts_manager.cc:194] Registered new tserver with Master: 1703837391ff42538d6bd508830992a1 (127.14.33.2:44475)
I20260925 05:05:54.405995 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19272
I20260925 05:05:54.406075 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:54.406205 14468 external_mini_cluster.cc:1428] Setting key 2ef7d34c9756761ec6c1e8534f7225b0
I20260925 05:05:54.407069 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38337
--builtin_ntp_servers=127.14.33.20:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 05:05:54.483361 19403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:54.483537 19403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:54.483557 19403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:54.484625 19403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:54.484660 19403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:54.484684 19403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:54.485973 19403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:54.486140 19403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:54.486361 19403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:54.490908 19411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:54.490899 19409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.490937 19403 server_base.cc:1034] running on GCE node
W20260925 05:05:54.491055 19408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:54.491314 19403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:54.491494 19403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:54.492622 19403 hybrid_clock.cc:648] HybridClock initialized: now 1790312754492607 us; error 29 us; skew 500 ppm
I20260925 05:05:54.493952 19403 webserver.cc:460] Webserver started at http://127.14.33.3:45065/ using document root <none> and password file <none>
I20260925 05:05:54.494180 19403 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:54.494232 19403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:54.494371 19403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:54.495338 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "4ebc5031a9ce472ba617cbd126bd590b"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "1a362888f304dfe9a292b1a8c4dd7b1c"
server_key_iv: "ffb84d0f53848968a84b985dec286760"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.495684 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "4ebc5031a9ce472ba617cbd126bd590b"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "1a362888f304dfe9a292b1a8c4dd7b1c"
server_key_iv: "ffb84d0f53848968a84b985dec286760"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.496826 19403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:54.497618 19417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.497857 19403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.497949 19403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "4ebc5031a9ce472ba617cbd126bd590b"
format_stamp: "Formatted at 2026-09-25 05:05:54 on dist-test-slave-4dmq"
server_key: "1a362888f304dfe9a292b1a8c4dd7b1c"
server_key_iv: "ffb84d0f53848968a84b985dec286760"
server_key_version: "encryptionkey@0"
I20260925 05:05:54.498034 19403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:54.501194 19403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:54.501431 19403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:54.501559 19403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:54.501754 19403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:54.502029 19403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:54.502079 19403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.502115 19403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:54.502141 19403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:54.507514 19403 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:45233
I20260925 05:05:54.507546 19530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:45233 every 8 connection(s)
I20260925 05:05:54.507846 19403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:54.510526 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.508516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:56195 (local address 127.14.33.62:38337)
0925 05:05:54.508589 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.508591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.508775 (+ 184us) server_negotiation.cc:407] Connection header received
0925 05:05:54.509317 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.509318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.509323 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.509362 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.510346 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.510347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.510347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.510351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.510365 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.510369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.510413 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.510466 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.510478 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:05:54.510672 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.508482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38337 (local address 127.14.33.3:56195)
0925 05:05:54.508666 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.508697 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.509066 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.509416 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.509419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.509426 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.510219 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.510219 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.510243 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.510267 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.510411 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.510414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.510416 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.510464 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.510504 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260925 05:05:54.511173 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19403
I20260925 05:05:54.511220 19531 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38337
I20260925 05:05:54.511286 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:54.511324 19531 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:54.511454 19531 heartbeater.cc:511] Master 127.14.33.62:38337 requested a full tablet report, sending...
I20260925 05:05:54.511459 14468 external_mini_cluster.cc:1428] Setting key 301c02a2d92ef5c388b89b82eef75136
I20260925 05:05:54.511787 19081 ts_manager.cc:194] Registered new tserver with Master: 4ebc5031a9ce472ba617cbd126bd590b (127.14.33.3:45233)
I20260925 05:05:54.512382 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:54.514465 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.513588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.14.33.62:38337)
0925 05:05:54.513640 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.513641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.513744 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:05:54.513818 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.513819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.513822 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.513873 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.514315 (+ 442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.514316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.514316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.514319 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.514330 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.514333 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.514382 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.514412 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.514422 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:05:54.515962 19081 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:05:54.517421 19081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:05:54.520556 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.519302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45233 (local address 127.0.0.1:47586)
0925 05:05:54.519423 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.519458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.519508 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.519770 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.519776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.519782 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.520189 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.520190 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.520205 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.520223 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.520407 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.520414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.520416 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.520479 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.520492 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260925 05:05:54.520572 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.519503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.14.33.3:45233)
0925 05:05:54.519624 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.519629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.519638 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:05:54.519659 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.519661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.519669 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.519742 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.520298 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.520304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.520304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.520308 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.520340 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.520359 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.520418 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.520483 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.520496 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260925 05:05:54.521332 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.519282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44475 (local address 127.0.0.1:43706)
0925 05:05:54.519423 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.519458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.519507 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.520071 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.520071 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.520074 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.520949 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.520950 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.520956 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.520967 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.521199 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.521200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.521200 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.521258 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.521271 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":46,"threads_started":1}
I20260925 05:05:54.521371 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.519358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.14.33.2:44475)
0925 05:05:54.519670 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.519674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.519683 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:05:54.519722 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.519725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.519733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.519797 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.521046 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.521055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.521055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.521060 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.521093 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.521140 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.521204 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.521273 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.521288 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":71,"threads_started":1}
I20260925 05:05:54.521310 19465 tablet_service.cc:1467] Processing CreateTablet for tablet a816e5d0d8594a748fbdee44fd440d34 (DEFAULT_TABLE table=test-table [id=c1907f2f6faa463bb7d7a5e6c016e72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:54.521582 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.519642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.14.33.1:44769)
0925 05:05:54.520037 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.520041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.520348 (+ 307us) server_negotiation.cc:407] Connection header received
0925 05:05:54.520378 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.520381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.520390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.520455 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.521227 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.521234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.521235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.521239 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.521273 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.521295 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.521350 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.521483 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.521498 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":95,"threads_started":1}
I20260925 05:05:54.521629 19465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816e5d0d8594a748fbdee44fd440d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:54.521746 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.519472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44769 (local address 127.0.0.1:57664)
0925 05:05:54.520268 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.520276 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.520320 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.520721 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.520724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.520731 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.521133 (+ 402us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.521134 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.521139 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.521149 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.521376 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.521377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.521377 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.521664 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.521678 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":61,"threads_started":1}
I20260925 05:05:54.522403 19203 tablet_service.cc:1467] Processing CreateTablet for tablet a816e5d0d8594a748fbdee44fd440d34 (DEFAULT_TABLE table=test-table [id=c1907f2f6faa463bb7d7a5e6c016e72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:54.522720 19203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816e5d0d8594a748fbdee44fd440d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:54.524575 19545 tablet_bootstrap.cc:492] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Bootstrap starting.
I20260925 05:05:54.525159 19545 tablet_bootstrap.cc:654] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:54.525389 19545 log.cc:824] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:54.525866 19547 tablet_bootstrap.cc:492] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: Bootstrap starting.
I20260925 05:05:54.526104 19545 tablet_bootstrap.cc:492] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: No bootstrap required, opened a new log
I20260925 05:05:54.526190 19545 ts_tablet_manager.cc:1397] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:54.526373 19334 tablet_service.cc:1467] Processing CreateTablet for tablet a816e5d0d8594a748fbdee44fd440d34 (DEFAULT_TABLE table=test-table [id=c1907f2f6faa463bb7d7a5e6c016e72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:05:54.526531 19547 tablet_bootstrap.cc:654] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:54.526675 19334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816e5d0d8594a748fbdee44fd440d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:54.526835 19547 log.cc:824] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:54.527593 19547 tablet_bootstrap.cc:492] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: No bootstrap required, opened a new log
I20260925 05:05:54.527661 19547 ts_tablet_manager.cc:1397] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:54.528766 19545 raft_consensus.cc:348] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.528924 19545 raft_consensus.cc:374] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:54.528957 19545 raft_consensus.cc:729] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebc5031a9ce472ba617cbd126bd590b, State: Initialized, Role: FOLLOWER
I20260925 05:05:54.529086 19545 consensus_queue.cc:260] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.529328 19549 tablet_bootstrap.cc:492] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Bootstrap starting.
I20260925 05:05:54.529351 19545 ts_tablet_manager.cc:1428] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 05:05:54.529480 19531 heartbeater.cc:503] Master 127.14.33.62:38337 was elected leader, sending a full tablet report...
I20260925 05:05:54.529951 19549 tablet_bootstrap.cc:654] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:54.530189 19547 raft_consensus.cc:348] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.530351 19547 raft_consensus.cc:374] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:54.530392 19547 raft_consensus.cc:729] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0584cf715144449cbee16830dccf034e, State: Initialized, Role: FOLLOWER
I20260925 05:05:54.530380 19549 log.cc:824] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:54.530498 19547 consensus_queue.cc:260] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.530733 19547 ts_tablet_manager.cc:1428] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:05:54.530797 19269 heartbeater.cc:503] Master 127.14.33.62:38337 was elected leader, sending a full tablet report...
I20260925 05:05:54.531039 19549 tablet_bootstrap.cc:492] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: No bootstrap required, opened a new log
I20260925 05:05:54.531104 19549 ts_tablet_manager.cc:1397] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:05:54.532858 19549 raft_consensus.cc:348] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.532966 19549 raft_consensus.cc:374] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:54.532985 19549 raft_consensus.cc:729] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1703837391ff42538d6bd508830992a1, State: Initialized, Role: FOLLOWER
I20260925 05:05:54.533077 19549 consensus_queue.cc:260] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.533291 19549 ts_tablet_manager.cc:1428] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:05:54.533385 19400 heartbeater.cc:503] Master 127.14.33.62:38337 was elected leader, sending a full tablet report...
W20260925 05:05:54.533840 19270 tablet.cc:2369] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:54.648526 19401 tablet.cc:2369] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:05:54.758353 19532 tablet.cc:2369] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:05:54.909950 19552 raft_consensus.cc:482] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:05:54.910066 19552 raft_consensus.cc:504] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.910413 19552 leader_election.cc:288] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1703837391ff42538d6bd508830992a1 (127.14.33.2:44475), 4ebc5031a9ce472ba617cbd126bd590b (127.14.33.3:45233)
I20260925 05:05:54.911738 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.910571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:45233 (local address 127.14.33.1:54903)
0925 05:05:54.910738 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.910747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.910796 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.910937 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.910939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.910942 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.911376 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.911377 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.911384 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.911395 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.911555 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.911556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.911557 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.911605 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.911619 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260925 05:05:54.911795 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.910645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:54903 (local address 127.14.33.3:45233)
0925 05:05:54.910692 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.910695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.910828 (+ 133us) server_negotiation.cc:407] Connection header received
0925 05:05:54.910858 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.910860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.910866 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.910922 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.911474 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.911475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.911475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.911479 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.911495 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.911499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.911554 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.911705 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.911718 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:05:54.911863 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.910669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:54409 (local address 127.14.33.2:44475)
0925 05:05:54.910714 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.910716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.910852 (+ 136us) server_negotiation.cc:407] Connection header received
0925 05:05:54.910891 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.910908 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.910914 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.910970 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.911606 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.911607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.911608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.911612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.911625 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.911629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.911679 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.911760 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.911774 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:05:54.912104 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.910574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44475 (local address 127.14.33.1:54409)
0925 05:05:54.910749 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.910766 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:54.910841 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:54.910985 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:54.910986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:54.910997 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:54.911530 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:05:54.911531 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:54.911542 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:05:54.911553 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:54.911684 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:54.911684 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:54.911685 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:05:54.911744 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:05:54.911770 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":59,"threads_started":1}
I20260925 05:05:54.912086 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816e5d0d8594a748fbdee44fd440d34" candidate_uuid: "0584cf715144449cbee16830dccf034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebc5031a9ce472ba617cbd126bd590b" is_pre_election: true
I20260925 05:05:54.912262 19485 raft_consensus.cc:2454] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0584cf715144449cbee16830dccf034e in term 0.
I20260925 05:05:54.912434 19354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816e5d0d8594a748fbdee44fd440d34" candidate_uuid: "0584cf715144449cbee16830dccf034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1703837391ff42538d6bd508830992a1" is_pre_election: true
I20260925 05:05:54.912477 19156 leader_election.cc:302] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0584cf715144449cbee16830dccf034e, 4ebc5031a9ce472ba617cbd126bd590b; no voters:
I20260925 05:05:54.912588 19354 raft_consensus.cc:2454] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0584cf715144449cbee16830dccf034e in term 0.
I20260925 05:05:54.912581 19552 raft_consensus.cc:2781] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:05:54.912645 19552 raft_consensus.cc:482] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:05:54.912708 19552 raft_consensus.cc:3037] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:54.913499 19552 raft_consensus.cc:504] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.913633 19552 leader_election.cc:288] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [CANDIDATE]: Term 1 election: Requested vote from peers 1703837391ff42538d6bd508830992a1 (127.14.33.2:44475), 4ebc5031a9ce472ba617cbd126bd590b (127.14.33.3:45233)
I20260925 05:05:54.913769 19354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816e5d0d8594a748fbdee44fd440d34" candidate_uuid: "0584cf715144449cbee16830dccf034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1703837391ff42538d6bd508830992a1"
I20260925 05:05:54.913797 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816e5d0d8594a748fbdee44fd440d34" candidate_uuid: "0584cf715144449cbee16830dccf034e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebc5031a9ce472ba617cbd126bd590b"
I20260925 05:05:54.913851 19354 raft_consensus.cc:3037] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:54.913861 19485 raft_consensus.cc:3037] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:54.914714 19485 raft_consensus.cc:2454] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0584cf715144449cbee16830dccf034e in term 1.
I20260925 05:05:54.914712 19354 raft_consensus.cc:2454] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0584cf715144449cbee16830dccf034e in term 1.
I20260925 05:05:54.914862 19156 leader_election.cc:302] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0584cf715144449cbee16830dccf034e, 4ebc5031a9ce472ba617cbd126bd590b; no voters:
I20260925 05:05:54.914992 19552 raft_consensus.cc:2781] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:54.915117 19552 raft_consensus.cc:686] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 1 LEADER]: Becoming Leader. State: Replica: 0584cf715144449cbee16830dccf034e, State: Running, Role: LEADER
I20260925 05:05:54.915237 19552 consensus_queue.cc:237] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } }
I20260925 05:05:54.916049 19080 catalog_manager.cc:5495] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0584cf715144449cbee16830dccf034e (127.14.33.1). New cstate: current_term: 1 leader_uuid: "0584cf715144449cbee16830dccf034e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0584cf715144449cbee16830dccf034e" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44769 } health_report { overall_health: HEALTHY } } }
I20260925 05:05:54.963254 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.962384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.14.33.1:44769)
0925 05:05:54.962422 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:54.962424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:54.962456 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:05:54.962483 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:54.962484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:54.962495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:54.962531 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:54.963088 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:54.963089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:54.963089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:54.963092 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:54.963103 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:54.963106 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:54.963148 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:54.963198 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 05:05:54.963208 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:05:54.964999 19485 raft_consensus.cc:1261] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 1 FOLLOWER]: Refusing update from remote peer 0584cf715144449cbee16830dccf034e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:54.965014 19354 raft_consensus.cc:1261] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 1 FOLLOWER]: Refusing update from remote peer 0584cf715144449cbee16830dccf034e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:05:54.965335 19552 consensus_queue.cc:1035] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebc5031a9ce472ba617cbd126bd590b" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:54.965490 19556 consensus_queue.cc:1035] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1703837391ff42538d6bd508830992a1" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:05:54.968577 19561 mvcc.cc:205] Tried to move back new op lower bound from 7333121044333940736 to 7333121044133326848. Current Snapshot: MvccSnapshot[applied={T|T < 7333121044333940736 or (T in {7333121044333940736})}]
I20260925 05:05:54.969354 19560 mvcc.cc:205] Tried to move back new op lower bound from 7333121044333940736 to 7333121044133326848. Current Snapshot: MvccSnapshot[applied={T|T < 7333121044333940736 or (T in {7333121044333940736})}]
I20260925 05:05:54.969930 19081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:54.970037 19081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:05:54.970788 19081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf24cb3a43df4175a527045d768a12a8: Sending DeleteTablet for 3 replicas of tablet a816e5d0d8594a748fbdee44fd440d34
I20260925 05:05:54.971098 19465 tablet_service.cc:1514] Processing DeleteTablet for tablet a816e5d0d8594a748fbdee44fd440d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:54 UTC) from {username='slave'} at 127.0.0.1:47586
I20260925 05:05:54.971191 19334 tablet_service.cc:1514] Processing DeleteTablet for tablet a816e5d0d8594a748fbdee44fd440d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:54 UTC) from {username='slave'} at 127.0.0.1:43706
I20260925 05:05:54.971300 19203 tablet_service.cc:1514] Processing DeleteTablet for tablet a816e5d0d8594a748fbdee44fd440d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:05:54 UTC) from {username='slave'} at 127.0.0.1:57664
I20260925 05:05:54.971319 19568 tablet_replica.cc:330] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: stopping tablet replica
I20260925 05:05:54.971426 19568 raft_consensus.cc:2229] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:54.971521 19568 raft_consensus.cc:2258] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:54.971657 19570 tablet_replica.cc:330] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e: stopping tablet replica
I20260925 05:05:54.971679 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19141
I20260925 05:05:54.971766 19570 raft_consensus.cc:2229] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:05:54.971850 19570 raft_consensus.cc:2258] T a816e5d0d8594a748fbdee44fd440d34 P 0584cf715144449cbee16830dccf034e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:54.971971 19569 tablet_replica.cc:330] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: stopping tablet replica
I20260925 05:05:54.972046 19569 raft_consensus.cc:2229] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:05:54.972136 19569 raft_consensus.cc:2258] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:05:54.972620 19569 ts_tablet_manager.cc:1905] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:54.973769 19568 ts_tablet_manager.cc:1905] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:05:54.974202 19569 ts_tablet_manager.cc:1918] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:54.974288 19569 log.cc:1196] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/a816e5d0d8594a748fbdee44fd440d34
I20260925 05:05:54.974514 19569 ts_tablet_manager.cc:1939] T a816e5d0d8594a748fbdee44fd440d34 P 1703837391ff42538d6bd508830992a1: Deleting consensus metadata
I20260925 05:05:54.974889 19067 catalog_manager.cc:4841] TS 1703837391ff42538d6bd508830992a1 (127.14.33.2:44475): tablet a816e5d0d8594a748fbdee44fd440d34 (table test-table [id=c1907f2f6faa463bb7d7a5e6c016e72c]) successfully deleted
I20260925 05:05:54.975193 19568 ts_tablet_manager.cc:1918] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:05:54.975255 19568 log.cc:1196] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/a816e5d0d8594a748fbdee44fd440d34
I20260925 05:05:54.975457 19568 ts_tablet_manager.cc:1939] T a816e5d0d8594a748fbdee44fd440d34 P 4ebc5031a9ce472ba617cbd126bd590b: Deleting consensus metadata
I20260925 05:05:54.975813 19065 catalog_manager.cc:4841] TS 4ebc5031a9ce472ba617cbd126bd590b (127.14.33.3:45233): tablet a816e5d0d8594a748fbdee44fd440d34 (table test-table [id=c1907f2f6faa463bb7d7a5e6c016e72c]) successfully deleted
W20260925 05:05:54.977005 19065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:44769 (error 108)
I20260925 05:05:54.977473 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:54.977354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44769 (local address 127.0.0.1:57688)
0925 05:05:54.977402 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:54.977440 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 05:05:54.977481 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19272
W20260925 05:05:54.977977 19065 catalog_manager.cc:4584] TS 0584cf715144449cbee16830dccf034e (127.14.33.1:44769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a816e5d0d8594a748fbdee44fd440d34: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44769: connect: Connection refused (error 111)
I20260925 05:05:54.981696 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19403
I20260925 05:05:54.986594 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19050
2026-09-25T05:05:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1003 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 05:05:55.005867 14468 mini_oidc.cc:152] Starting JWKS server
I20260925 05:05:55.006908 14468 webserver.cc:460] Webserver started at https://127.0.0.1:38175/ using document root <none> and password file <none>
I20260925 05:05:55.007014 14468 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:05:55.007207 14468 webserver.cc:460] Webserver started at http://127.0.0.1:38035/ using document root <none> and password file <none>
2026-09-25T05:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:55Z Disabled control of system clock
I20260925 05:05:55.014940 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:43523
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:43523
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38175/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/testchainca.cert with env {}
W20260925 05:05:55.090847 19577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:55.091022 19577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:55.091041 19577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:55.092211 19577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:55.092252 19577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:55.092271 19577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:55.092307 19577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:55.092338 19577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:05:55.092355 19577 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:05:55.092376 19577 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38175/jwks.json
I20260925 05:05:55.093930 19577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:43523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38175/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:55.094170 19577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:55.094398 19577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:55.098974 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.098975 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.099017 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:55.099345 19577 server_base.cc:1034] running on GCE node
I20260925 05:05:55.099656 19577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:55.099974 19577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:55.101174 19577 hybrid_clock.cc:648] HybridClock initialized: now 1790312755101138 us; error 45 us; skew 500 ppm
I20260925 05:05:55.102365 19577 webserver.cc:460] Webserver started at http://127.14.33.62:42105/ using document root <none> and password file <none>
I20260925 05:05:55.102619 19577 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:55.102706 19577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:55.102859 19577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:55.103933 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "a2386bbe19c749c1aeb65c081701afb9"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "8861f08536a22eb4e7e9015ec27c983c"
server_key_iv: "999641f514f87bc93b2d4b8d5426171d"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.104316 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "a2386bbe19c749c1aeb65c081701afb9"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "8861f08536a22eb4e7e9015ec27c983c"
server_key_iv: "999641f514f87bc93b2d4b8d5426171d"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.105791 19577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:55.106482 19591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:55.106664 19577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:55.106774 19577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "a2386bbe19c749c1aeb65c081701afb9"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "8861f08536a22eb4e7e9015ec27c983c"
server_key_iv: "999641f514f87bc93b2d4b8d5426171d"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.106881 19577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:55.133257 19577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:55.133538 19577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:55.133689 19577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:55.137411 19577 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:43523
I20260925 05:05:55.137458 19643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:43523 every 8 connection(s)
I20260925 05:05:55.137742 19577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:55.138538 19644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:55.138837 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19577
I20260925 05:05:55.138965 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:55.139142 14468 external_mini_cluster.cc:1428] Setting key a24bdaaf1c88049ecdc32b74e856b216
I20260925 05:05:55.141031 19644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9: Bootstrap starting.
I20260925 05:05:55.141635 19644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:55.141923 19644 log.cc:824] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:55.142557 19644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9: No bootstrap required, opened a new log
I20260925 05:05:55.144336 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.139612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.14.33.62:43523)
0925 05:05:55.139828 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.139835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.139858 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:05:55.140252 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.140258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.140314 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.140463 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:55.140655 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.141592 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.142483 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.142925 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.143072 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.143708 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:55.143712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:55.143712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:55.143899 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:55.143928 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:55.143938 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:55.144006 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:55.144086 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:05:55.144157 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260925 05:05:55.144572 19644 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2386bbe19c749c1aeb65c081701afb9" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43523 } }
I20260925 05:05:55.144744 19644 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:55.144797 19644 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2386bbe19c749c1aeb65c081701afb9, State: Initialized, Role: FOLLOWER
I20260925 05:05:55.144910 19644 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2386bbe19c749c1aeb65c081701afb9" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43523 } }
I20260925 05:05:55.145004 19644 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:55.145038 19644 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:55.145094 19644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:55.145213 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:55.145663 19644 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2386bbe19c749c1aeb65c081701afb9" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43523 } }
I20260925 05:05:55.145805 19644 leader_election.cc:302] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2386bbe19c749c1aeb65c081701afb9; no voters:
I20260925 05:05:55.145959 19644 leader_election.cc:288] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:55.145984 19649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:55.146088 19649 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [term 1 LEADER]: Becoming Leader. State: Replica: a2386bbe19c749c1aeb65c081701afb9, State: Running, Role: LEADER
I20260925 05:05:55.146167 19649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2386bbe19c749c1aeb65c081701afb9" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43523 } }
I20260925 05:05:55.146224 19644 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:55.146495 19650 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2386bbe19c749c1aeb65c081701afb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2386bbe19c749c1aeb65c081701afb9" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43523 } } }
I20260925 05:05:55.146582 19650 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.146746 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2386bbe19c749c1aeb65c081701afb9. Latest consensus state: current_term: 1 leader_uuid: "a2386bbe19c749c1aeb65c081701afb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2386bbe19c749c1aeb65c081701afb9" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43523 } } }
I20260925 05:05:55.146803 19654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:55.146811 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.147428 19654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:55.148885 19654 catalog_manager.cc:1347] Generated new cluster ID: 6a68311369084f7cb9bc957f16d134cb
I20260925 05:05:55.148929 19654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:55.156852 19654 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:55.157656 19654 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:55.162760 19654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2386bbe19c749c1aeb65c081701afb9: Generated new TSK 0
I20260925 05:05:55.162901 19654 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 05:05:55.190918 19674 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 05:05:55.191004 19674 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:05:55.191241 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.176333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.14.33.62:43523)
0925 05:05:55.176391 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.176393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.176506 (+ 113us) server_negotiation.cc:407] Connection header received
0925 05:05:55.176566 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.176567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.176578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.176650 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:05:55.176784 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.177566 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.178299 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.178604 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.178673 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.178853 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:05:55.191037 (+ 12184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:05:55.191173 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 05:05:55.191305 19647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 05:05:55.191571 19673 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:55.176253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:43523 (local address 127.0.0.1:44036)
0925 05:05:55.176442 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:55.176451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:55.176504 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:55.176660 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:55.176661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:55.176662 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:05:55.176723 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.176725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.177644 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.177645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.178214 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.178215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.178690 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.178690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.178732 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.178749 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:05:55.191168 (+ 12419us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 05:05:55.191254 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.33.62:43523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.33.62:43523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38175/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 05:05:55.192054 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19577
2026-09-25T05:05:55Z chronyd exiting
I20260925 05:05:55.227269 14468 test_util.cc:175] -----------------------------------------------
I20260925 05:05:55.227329 14468 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312729583056-14468-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 05:05:55.228070 14468 mini_oidc.cc:152] Starting JWKS server
I20260925 05:05:55.228711 14468 webserver.cc:460] Webserver started at https://127.0.0.1:45551/ using document root <none> and password file <none>
I20260925 05:05:55.228747 14468 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:05:55.228920 14468 webserver.cc:460] Webserver started at http://127.0.0.1:45423/ using document root <none> and password file <none>
2026-09-25T05:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:55Z Disabled control of system clock
I20260925 05:05:55.236088 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36409
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:36409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45551/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/testchainca.cert with env {}
W20260925 05:05:55.313107 19680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:55.313328 19680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:55.313364 19680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:55.314455 19680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:55.314514 19680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:55.314535 19680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:55.314556 19680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:55.314584 19680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:05:55.314608 19680 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:05:55.314636 19680 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45551/jwks.json
I20260925 05:05:55.316048 19680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:36409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45551/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:55.316298 19680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:55.316504 19680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:55.321027 19686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.321089 19688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.321035 19685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:55.321405 19680 server_base.cc:1034] running on GCE node
I20260925 05:05:55.321628 19680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:55.321887 19680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:55.323041 19680 hybrid_clock.cc:648] HybridClock initialized: now 1790312755323031 us; error 26 us; skew 500 ppm
I20260925 05:05:55.324141 19680 webserver.cc:460] Webserver started at http://127.14.33.62:40319/ using document root <none> and password file <none>
I20260925 05:05:55.324369 19680 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:55.324433 19680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:55.324571 19680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:55.325590 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "38c8e4d02639491685272daec421eaf3"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "7a673b6cd2aa3c77e2409e066e97ad64"
server_key_iv: "6a4b0273b5b4855f6281009b13936802"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.325930 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "38c8e4d02639491685272daec421eaf3"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "7a673b6cd2aa3c77e2409e066e97ad64"
server_key_iv: "6a4b0273b5b4855f6281009b13936802"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.327091 19680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:55.327701 19694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:55.327853 19680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:55.327939 19680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "38c8e4d02639491685272daec421eaf3"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "7a673b6cd2aa3c77e2409e066e97ad64"
server_key_iv: "6a4b0273b5b4855f6281009b13936802"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.328008 19680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:55.344866 19680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:55.345188 19680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:55.345327 19680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:55.348892 19680 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:36409
I20260925 05:05:55.348933 19746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:36409 every 8 connection(s)
I20260925 05:05:55.349291 19680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:55.349514 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19680
I20260925 05:05:55.349625 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:55.349788 14468 external_mini_cluster.cc:1428] Setting key 504d1146f880165dc86ab42c44bd874e
I20260925 05:05:55.349975 19747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:55.352988 19747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3: Bootstrap starting.
I20260925 05:05:55.353593 19747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:55.353885 19747 log.cc:824] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:55.354547 19747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3: No bootstrap required, opened a new log
I20260925 05:05:55.355988 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.350604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.14.33.62:36409)
0925 05:05:55.350817 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.350825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.350855 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:05:55.351334 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.351343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.351420 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.351563 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:55.351720 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.353036 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.354178 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.354634 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.354750 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.355391 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:55.355394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:55.355395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:55.355547 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:55.355574 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:55.355584 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:55.355656 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:55.355734 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:05:55.355813 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260925 05:05:55.356570 19747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c8e4d02639491685272daec421eaf3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36409 } }
I20260925 05:05:55.356714 19747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:55.356782 19747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c8e4d02639491685272daec421eaf3, State: Initialized, Role: FOLLOWER
I20260925 05:05:55.356921 19747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c8e4d02639491685272daec421eaf3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36409 } }
I20260925 05:05:55.356992 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:55.357004 19747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:55.357043 19747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:55.357096 19747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:55.357599 19747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c8e4d02639491685272daec421eaf3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36409 } }
I20260925 05:05:55.357718 19747 leader_election.cc:302] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38c8e4d02639491685272daec421eaf3; no voters:
I20260925 05:05:55.357867 19747 leader_election.cc:288] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:55.357904 19752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:55.358034 19752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [term 1 LEADER]: Becoming Leader. State: Replica: 38c8e4d02639491685272daec421eaf3, State: Running, Role: LEADER
I20260925 05:05:55.358112 19747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:55.358110 19752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c8e4d02639491685272daec421eaf3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36409 } }
I20260925 05:05:55.358455 19754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38c8e4d02639491685272daec421eaf3. Latest consensus state: current_term: 1 leader_uuid: "38c8e4d02639491685272daec421eaf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c8e4d02639491685272daec421eaf3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36409 } } }
I20260925 05:05:55.358457 19753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38c8e4d02639491685272daec421eaf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c8e4d02639491685272daec421eaf3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36409 } } }
I20260925 05:05:55.358597 19753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.358582 19754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.359090 19760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:55.359587 19760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:55.360939 19760 catalog_manager.cc:1347] Generated new cluster ID: 4351b781ac054d7abee6dbdfc7f69619
I20260925 05:05:55.360988 19760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:55.372092 19760 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:55.372541 19760 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:55.378085 19760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38c8e4d02639491685272daec421eaf3: Generated new TSK 0
I20260925 05:05:55.378244 19760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:55.401082 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.387246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.14.33.62:36409)
0925 05:05:55.387305 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.387307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.387409 (+ 102us) server_negotiation.cc:407] Connection header received
0925 05:05:55.387476 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.387477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.387488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.387551 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:05:55.387691 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.388182 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.389346 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.389548 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.389609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.389778 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:05:55.400932 (+ 11154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:05:55.401031 (+ 99us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 05:05:55.401170 19750 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 05:05:55.401269 19776 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:55.387146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36409 (local address 127.0.0.1:55748)
0925 05:05:55.387332 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:55.387343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:55.387408 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:55.387571 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:55.387572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:55.387574 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:05:55.387629 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.387632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.388270 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.388271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.389245 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.389249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.389626 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.389627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.389687 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.389693 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:05:55.401028 (+ 11335us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 05:05:55.401162 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.33.62:36409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
E20260925 05:05:55.401340 19777 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 05:05:55.401379 19777 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 05:05:55.401518 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19680
2026-09-25T05:05:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 05:05:55.438901 14468 mini_oidc.cc:152] Starting JWKS server
I20260925 05:05:55.439522 14468 webserver.cc:460] Webserver started at https://127.0.0.1:45405/ using document root <none> and password file <none>
I20260925 05:05:55.439560 14468 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 05:05:55.439704 14468 webserver.cc:460] Webserver started at http://127.0.0.1:38277/ using document root <none> and password file <none>
2026-09-25T05:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:55Z Disabled control of system clock
I20260925 05:05:55.446851 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:35867
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:41405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:35867
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:05:55.524603 19783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:55.524804 19783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:55.524837 19783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:55.525991 19783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:55.526054 19783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:55.526074 19783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:55.526093 19783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:55.526124 19783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:05:55.526152 19783 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:05:55.526182 19783 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45405/jwks.json
I20260925 05:05:55.527453 19783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:41405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:35867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:55.527693 19783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:55.527949 19783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:05:55.532544 19783 server_base.cc:1034] running on GCE node
W20260925 05:05:55.532516 19791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.532563 19788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.532516 19789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:55.532882 19783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:55.533093 19783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:55.534310 19783 hybrid_clock.cc:648] HybridClock initialized: now 1790312755534300 us; error 25 us; skew 500 ppm
I20260925 05:05:55.535470 19783 webserver.cc:460] Webserver started at http://127.14.33.62:40315/ using document root <none> and password file <none>
I20260925 05:05:55.535669 19783 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:55.535710 19783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:55.535858 19783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:55.536777 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "687d961c01014556b7cd312e0a08eaac"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "bc3086448147ae003ac7a3e3b4b25dde"
server_key_iv: "cc1bd0fd20894012de2c49d499d6c17a"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.537089 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "687d961c01014556b7cd312e0a08eaac"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "bc3086448147ae003ac7a3e3b4b25dde"
server_key_iv: "cc1bd0fd20894012de2c49d499d6c17a"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.538322 19783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:55.538990 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:55.539170 19783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:55.539273 19783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "687d961c01014556b7cd312e0a08eaac"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "bc3086448147ae003ac7a3e3b4b25dde"
server_key_iv: "cc1bd0fd20894012de2c49d499d6c17a"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.539345 19783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:55.577991 19783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:55.578323 19783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:55.578477 19783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:55.582270 19783 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:35867
I20260925 05:05:55.582329 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:35867 every 8 connection(s)
I20260925 05:05:55.582654 19783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:55.583330 19850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:55.585937 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac: Bootstrap starting.
I20260925 05:05:55.586460 19850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:55.586694 19850 log.cc:824] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:55.587285 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac: No bootstrap required, opened a new log
I20260925 05:05:55.588757 19850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d961c01014556b7cd312e0a08eaac" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 35867 } }
I20260925 05:05:55.588884 19850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:55.588905 19850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687d961c01014556b7cd312e0a08eaac, State: Initialized, Role: FOLLOWER
I20260925 05:05:55.589008 19850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d961c01014556b7cd312e0a08eaac" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 35867 } }
I20260925 05:05:55.589056 19850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:55.589074 19850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:55.589104 19850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:55.589641 19850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d961c01014556b7cd312e0a08eaac" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 35867 } }
I20260925 05:05:55.589759 19850 leader_election.cc:302] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687d961c01014556b7cd312e0a08eaac; no voters:
I20260925 05:05:55.589907 19850 leader_election.cc:288] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:55.589954 19853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:55.590108 19853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [term 1 LEADER]: Becoming Leader. State: Replica: 687d961c01014556b7cd312e0a08eaac, State: Running, Role: LEADER
I20260925 05:05:55.590152 19850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:55.590221 19853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d961c01014556b7cd312e0a08eaac" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 35867 } }
I20260925 05:05:55.590685 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19783
I20260925 05:05:55.590785 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:55.590948 14468 external_mini_cluster.cc:1428] Setting key 961aac6eab6d842a10ed89c99e9877f4
I20260925 05:05:55.591347 19854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687d961c01014556b7cd312e0a08eaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d961c01014556b7cd312e0a08eaac" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 35867 } } }
I20260925 05:05:55.591447 19854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.591603 19855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687d961c01014556b7cd312e0a08eaac. Latest consensus state: current_term: 1 leader_uuid: "687d961c01014556b7cd312e0a08eaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d961c01014556b7cd312e0a08eaac" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 35867 } } }
I20260925 05:05:55.591693 19855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.591825 19867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:55.592407 19867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:55.593907 19867 catalog_manager.cc:1347] Generated new cluster ID: 5a70a9b98ce540ef812074a5f1ba8f5e
I20260925 05:05:55.593958 19867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:55.597206 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.591471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.14.33.62:35867)
0925 05:05:55.591709 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.591719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.592105 (+ 386us) server_negotiation.cc:407] Connection header received
0925 05:05:55.592229 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.592236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.592304 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.592509 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:55.592668 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.593954 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.595094 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.595675 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.595777 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.596463 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:55.596467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:55.596467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:55.596678 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:55.596719 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:55.596744 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:55.596822 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:55.596920 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 05:05:55.596989 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260925 05:05:55.598042 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:55.607963 19867 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:55.608415 19867 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:55.616024 19867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687d961c01014556b7cd312e0a08eaac: Generated new TSK 0
I20260925 05:05:55.616186 19867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:55.629087 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.625632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.14.33.62:35867)
0925 05:05:55.625681 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.625683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.625799 (+ 116us) server_negotiation.cc:407] Connection header received
0925 05:05:55.625866 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.625867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.625877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.625935 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:05:55.626078 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.626542 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.627668 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.627852 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.627897 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.628079 (+ 182us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:05:55.628901 (+ 822us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:05:55.629019 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 05:05:55.629217 19879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:55.625539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:35867 (local address 127.0.0.1:49084)
0925 05:05:55.625722 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:55.625733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:55.625800 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:55.625962 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:55.625963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:55.625965 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:05:55.626015 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.626017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.626626 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.626626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.627573 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.627576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.627933 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.627934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.627993 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.627998 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:05:55.629002 (+ 1004us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 05:05:55.629098 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.33.62:35867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
W20260925 05:05:55.629160 19868 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45405/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 05:05:55.629442 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19783
E20260925 05:05:55.639406 19880 webserver.cc:533] Webserver: sslize failed:
E20260925 05:05:55.639456 19880 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T05:05:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T05:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:55Z Disabled control of system clock
I20260925 05:05:55.674039 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:46029
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:46029
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/testchainca.cert with env {}
W20260925 05:05:55.752640 19884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:55.752856 19884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:55.752890 19884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:55.754096 19884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:55.754170 19884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:55.754197 19884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:55.754228 19884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:55.754263 19884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 05:05:55.754292 19884 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 05:05:55.754317 19884 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 05:05:55.755760 19884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:46029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:55.756012 19884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:55.756237 19884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:55.761003 19892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.761054 19889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:55.761010 19890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:55.761193 19884 server_base.cc:1034] running on GCE node
I20260925 05:05:55.761435 19884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:55.761708 19884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:55.762885 19884 hybrid_clock.cc:648] HybridClock initialized: now 1790312755762874 us; error 30 us; skew 500 ppm
I20260925 05:05:55.764137 19884 webserver.cc:460] Webserver started at http://127.14.33.62:40283/ using document root <none> and password file <none>
I20260925 05:05:55.764369 19884 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:55.764439 19884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:55.764590 19884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:55.765640 19884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "5a8e5861e36345cc85294aa210c56b5b"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "dfb8b2fbbc5ba3df3b0b39b8425567d7"
server_key_iv: "e0088ed0ff4c817c62d155e760e6467f"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.765998 19884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "5a8e5861e36345cc85294aa210c56b5b"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "dfb8b2fbbc5ba3df3b0b39b8425567d7"
server_key_iv: "e0088ed0ff4c817c62d155e760e6467f"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.767686 19884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:55.768473 19898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:55.768803 19884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:55.768927 19884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "5a8e5861e36345cc85294aa210c56b5b"
format_stamp: "Formatted at 2026-09-25 05:05:55 on dist-test-slave-4dmq"
server_key: "dfb8b2fbbc5ba3df3b0b39b8425567d7"
server_key_iv: "e0088ed0ff4c817c62d155e760e6467f"
server_key_version: "encryptionkey@0"
I20260925 05:05:55.769004 19884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:55.784529 19884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:55.784808 19884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:55.784962 19884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:55.788635 19884 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:46029
I20260925 05:05:55.788663 19950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:46029 every 8 connection(s)
I20260925 05:05:55.789000 19884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:55.789970 19951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:55.792050 19951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b: Bootstrap starting.
I20260925 05:05:55.792546 19951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:55.792769 19951 log.cc:824] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:55.793452 19951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b: No bootstrap required, opened a new log
I20260925 05:05:55.794979 19951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8e5861e36345cc85294aa210c56b5b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46029 } }
I20260925 05:05:55.795110 19951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:55.795131 19951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8e5861e36345cc85294aa210c56b5b, State: Initialized, Role: FOLLOWER
I20260925 05:05:55.795266 19951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8e5861e36345cc85294aa210c56b5b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46029 } }
I20260925 05:05:55.795346 19951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:55.795368 19951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:55.795395 19951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:55.795845 19951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8e5861e36345cc85294aa210c56b5b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46029 } }
I20260925 05:05:55.795924 19951 leader_election.cc:302] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a8e5861e36345cc85294aa210c56b5b; no voters:
I20260925 05:05:55.796039 19951 leader_election.cc:288] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:55.796110 19954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:55.796288 19954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [term 1 LEADER]: Becoming Leader. State: Replica: 5a8e5861e36345cc85294aa210c56b5b, State: Running, Role: LEADER
I20260925 05:05:55.796306 19951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:55.796387 19954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8e5861e36345cc85294aa210c56b5b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46029 } }
I20260925 05:05:55.796718 19955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a8e5861e36345cc85294aa210c56b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8e5861e36345cc85294aa210c56b5b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46029 } } }
I20260925 05:05:55.796820 19956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a8e5861e36345cc85294aa210c56b5b. Latest consensus state: current_term: 1 leader_uuid: "5a8e5861e36345cc85294aa210c56b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8e5861e36345cc85294aa210c56b5b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46029 } } }
I20260925 05:05:55.796842 19955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.796890 19956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:55.797047 19963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:55.797590 19963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:55.798049 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 19884
I20260925 05:05:55.798148 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:55.798333 14468 external_mini_cluster.cc:1428] Setting key f59298d1967189f511211392687f4dfd
I20260925 05:05:55.799199 19963 catalog_manager.cc:1347] Generated new cluster ID: 7f10250b2e594ec08f2ff0f4939984cd
I20260925 05:05:55.799264 19963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:55.805059 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.798753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.14.33.62:46029)
0925 05:05:55.799269 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.799278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.799475 (+ 197us) server_negotiation.cc:407] Connection header received
0925 05:05:55.799605 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.799612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.799676 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.799803 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:55.800021 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.801354 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.802479 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.803108 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.803219 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.804058 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:55.804063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:55.804063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:05:55.804289 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:55.804332 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:05:55.804347 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:55.804422 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:55.804517 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:05:55.804591 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":87,"threads_started":1}
I20260925 05:05:55.806365 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:05:55.811160 19963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:55.811838 19963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:55.815495 19963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a8e5861e36345cc85294aa210c56b5b: Generated new TSK 0
I20260925 05:05:55.815658 19963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:05:56.251705 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:55.823901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.14.33.62:46029)
0925 05:05:55.823957 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:55.823959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:55.824062 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:05:55.824097 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:55.824098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:55.824111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:55.824201 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 05:05:55.824326 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.825190 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.825936 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.826259 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.826328 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.826462 (+ 134us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 05:05:56.251494 (+425032us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:05:56.251647 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42198: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 05:05:56.251765 19973 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42198: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 05:05:56.251906 19980 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:55.823808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46029 (local address 127.0.0.1:42198)
0925 05:05:55.823983 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:55.823995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:55.824052 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:55.824204 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:55.824205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:55.824206 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 05:05:55.824254 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.824256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.825275 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.825275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.825856 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:55.825858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:55.826339 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:55.826339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:55.826377 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:55.826382 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 05:05:56.251630 (+425248us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 05:05:56.251748 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.33.62:46029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260925 05:05:56.252305 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 19884
2026-09-25T05:05:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (603 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (101 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (78 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:56 dist-test-slave-4dmq krb5kdc[19992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:56 dist-test-slave-4dmq krb5kdc[19992](info): set up 2 sockets
Sep 25 05:05:56 dist-test-slave-4dmq krb5kdc[19992](info): commencing operation
krb5kdc: starting...
W20260925 05:05:57.394289 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:05:57.505909 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:45643
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:45643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:57.580822 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:57.581073 20005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:57.581110 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:57.582273 20005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:57.582342 20005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:57.582363 20005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:57.582381 20005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:57.582410 20005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:57.583703 20005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:45643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:45643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:57.583976 20005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:57.584178 20005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:57.588884 20011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:57.588919 20010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:57.588927 20005 server_base.cc:1034] running on GCE node
W20260925 05:05:57.588901 20013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:57.589303 20005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:57.589560 20005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:57.590695 20005 hybrid_clock.cc:648] HybridClock initialized: now 1790312757590681 us; error 30 us; skew 500 ppm
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:57.592967 20005 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:05:57.593437 20005 webserver.cc:460] Webserver started at http://127.14.33.62:46555/ using document root <none> and password file <none>
I20260925 05:05:57.593650 20005 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:57.593699 20005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:57.593834 20005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:57.594796 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "c1c197ae130f410b8caae31ad0602f27"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "3203008c51d3ca7ee067b17f1afb355f"
server_key_iv: "e6c0b25cc1a7030e56c073d9065611a5"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.595177 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "c1c197ae130f410b8caae31ad0602f27"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "3203008c51d3ca7ee067b17f1afb355f"
server_key_iv: "e6c0b25cc1a7030e56c073d9065611a5"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.596407 20005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:05:57.597055 20020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.597287 20005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:57.597366 20005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "c1c197ae130f410b8caae31ad0602f27"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "3203008c51d3ca7ee067b17f1afb355f"
server_key_iv: "e6c0b25cc1a7030e56c073d9065611a5"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.597415 20005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:57.613809 20005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:57.615803 20005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:57.615902 20005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:57.619423 20005 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:45643
I20260925 05:05:57.619486 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:45643 every 8 connection(s)
I20260925 05:05:57.619872 20005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:57.620061 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20005
I20260925 05:05:57.620172 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:57.620337 14468 external_mini_cluster.cc:1428] Setting key 18292aa67bf9e054ca4d9b5530d11f75
I20260925 05:05:57.620572 20073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:57.623163 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27: Bootstrap starting.
I20260925 05:05:57.623685 20073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:57.623948 20073 log.cc:824] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:57.624540 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27: No bootstrap required, opened a new log
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:57.626291 20073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c197ae130f410b8caae31ad0602f27" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45643 } }
I20260925 05:05:57.626475 20073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:57.626546 20073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c197ae130f410b8caae31ad0602f27, State: Initialized, Role: FOLLOWER
I20260925 05:05:57.626667 20073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c197ae130f410b8caae31ad0602f27" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45643 } }
I20260925 05:05:57.626740 20073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:57.626781 20073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:57.626827 20073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:57.627310 20073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c197ae130f410b8caae31ad0602f27" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45643 } }
I20260925 05:05:57.627420 20073 leader_election.cc:302] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c197ae130f410b8caae31ad0602f27; no voters:
I20260925 05:05:57.627604 20073 leader_election.cc:288] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:57.627631 20078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:57.627781 20078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [term 1 LEADER]: Becoming Leader. State: Replica: c1c197ae130f410b8caae31ad0602f27, State: Running, Role: LEADER
I20260925 05:05:57.627848 20073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:57.627856 20078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c197ae130f410b8caae31ad0602f27" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45643 } }
I20260925 05:05:57.628333 20078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c197ae130f410b8caae31ad0602f27. Latest consensus state: current_term: 1 leader_uuid: "c1c197ae130f410b8caae31ad0602f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c197ae130f410b8caae31ad0602f27" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45643 } } }
I20260925 05:05:57.628337 20080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c197ae130f410b8caae31ad0602f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c197ae130f410b8caae31ad0602f27" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45643 } } }
I20260925 05:05:57.628423 20078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:57.628427 20080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27 [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:57.628674 20083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:57.629194 20083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:57.631217 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:57.620792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.14.33.62:45643)
0925 05:05:57.620957 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:57.620967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:57.620996 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:05:57.621635 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:57.621642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:57.621696 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:57.621832 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:57.622480 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.623281 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.624022 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.624392 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.624496 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:57.626860 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:57.626865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:57.626865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:57.626877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:57.629109 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:57.629686 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:57.629689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:57.629689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:57.629787 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:57.629992 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:57.629993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:57.629994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:57.630318 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:57.630471 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:57.630805 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 05:05:57.631036 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 05:05:57.637513 20083 catalog_manager.cc:1347] Generated new cluster ID: e1fd324d7d9c4e62b1188dae49cf0c7c
I20260925 05:05:57.637568 20083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:57.663265 20083 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:57.663918 20083 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:57.666203 20083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c197ae130f410b8caae31ad0602f27: Generated new TSK 0
I20260925 05:05:57.666375 20083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:05:57.695951 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45643
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:57.772207 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:57.772426 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:57.772462 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:57.773629 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:57.773700 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:57.773746 20099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:57.775171 20099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:57.775432 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:57.775671 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:57.780501 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:57.780508 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:57.780514 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:57.780704 20099 server_base.cc:1034] running on GCE node
I20260925 05:05:57.780889 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:57.781159 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:57.782315 20099 hybrid_clock.cc:648] HybridClock initialized: now 1790312757782293 us; error 38 us; skew 500 ppm
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:57.784852 20099 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:05:57.785336 20099 webserver.cc:460] Webserver started at http://127.14.33.1:33465/ using document root <none> and password file <none>
I20260925 05:05:57.785563 20099 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:57.785609 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:57.785691 20099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:57.786634 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "4aa8b4f4fd8d4988be0956848c922a42"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "7de4ab4aee17c8820ab6feef8ef584d5"
server_key_iv: "be3cd1cc45818af95bdb7a4d116f51c9"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.786931 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "4aa8b4f4fd8d4988be0956848c922a42"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "7de4ab4aee17c8820ab6feef8ef584d5"
server_key_iv: "be3cd1cc45818af95bdb7a4d116f51c9"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.788092 20099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:57.788694 20114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.788862 20099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:57.788959 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "4aa8b4f4fd8d4988be0956848c922a42"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "7de4ab4aee17c8820ab6feef8ef584d5"
server_key_iv: "be3cd1cc45818af95bdb7a4d116f51c9"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.789011 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:57.808313 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:57.810473 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:57.810578 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:57.810757 20099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:57.810986 20099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:57.811014 20099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.811033 20099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:57.811046 20099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.816361 20099 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:38013
I20260925 05:05:57.816489 20227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:38013 every 8 connection(s)
I20260925 05:05:57.816716 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:05:57.820770 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20099
I20260925 05:05:57.820878 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:57.821061 14468 external_mini_cluster.cc:1428] Setting key 57ce8160c43de2a8209cd4c5a4dfaeff
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:57.826737 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:57.817534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:46413 (local address 127.14.33.62:45643)
0925 05:05:57.817575 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:57.817577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:57.817779 (+ 202us) server_negotiation.cc:407] Connection header received
0925 05:05:57.818723 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:57.818724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:57.818735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:57.818803 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:57.819384 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.819917 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.821096 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.821444 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.821498 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:57.823918 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:57.823920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:57.823921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:57.823929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:57.825376 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:57.825793 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:57.825795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:57.825795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:57.825854 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:57.826107 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:57.826108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:57.826108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:57.826229 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:57.826300 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:57.826560 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 05:05:57.826649 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:05:57.826925 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:57.817501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45643 (local address 127.14.33.1:46413)
0925 05:05:57.817651 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:57.817685 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:57.818574 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:57.818854 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:57.818857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:57.819101 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:57.819294 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:57.819298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.820000 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.820001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:57.821001 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:57.821004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.821526 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.821527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:57.821664 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:57.822218 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:57.822229 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:57.823814 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:57.825475 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:57.825483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:57.825485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:57.825701 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:57.825924 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:57.825925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:57.825925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:57.826030 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:57.826303 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:57.826310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:57.826433 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 05:05:57.826539 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 05:05:57.826689 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":96,"threads_started":1}
I20260925 05:05:57.827416 20228 heartbeater.cc:347] Connected to a master server at 127.14.33.62:45643
I20260925 05:05:57.827525 20228 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:57.827716 20228 heartbeater.cc:511] Master 127.14.33.62:45643 requested a full tablet report, sending...
I20260925 05:05:57.828303 20037 ts_manager.cc:194] Registered new tserver with Master: 4aa8b4f4fd8d4988be0956848c922a42 (127.14.33.1:38013)
I20260925 05:05:57.828893 20037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:46413
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:05:57.845767 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45643
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:57.921744 20233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:57.921931 20233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:57.921952 20233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:57.923015 20233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:57.923053 20233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:57.923075 20233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:57.924396 20233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:57.924619 20233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:57.924813 20233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:57.929235 20238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:57.929302 20239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:57.929247 20241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:57.929694 20233 server_base.cc:1034] running on GCE node
I20260925 05:05:57.929914 20233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:57.930208 20233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:57.931401 20233 hybrid_clock.cc:648] HybridClock initialized: now 1790312757931379 us; error 34 us; skew 500 ppm
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:57.933743 20233 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:05:57.934082 20233 webserver.cc:460] Webserver started at http://127.14.33.2:45665/ using document root <none> and password file <none>
I20260925 05:05:57.934255 20233 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:57.934294 20233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:57.934374 20233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:57.935225 20233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "a85567cbea964a23b3b2fd3f70c63361"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "37589bf45586fccbeeef8bf2db10895a"
server_key_iv: "07717d3bb3465df4e4554fc5a410f841"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.935504 20233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "a85567cbea964a23b3b2fd3f70c63361"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "37589bf45586fccbeeef8bf2db10895a"
server_key_iv: "07717d3bb3465df4e4554fc5a410f841"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.936647 20233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:57.937278 20248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.937414 20233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:57.937487 20233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "a85567cbea964a23b3b2fd3f70c63361"
format_stamp: "Formatted at 2026-09-25 05:05:57 on dist-test-slave-4dmq"
server_key: "37589bf45586fccbeeef8bf2db10895a"
server_key_iv: "07717d3bb3465df4e4554fc5a410f841"
server_key_version: "encryptionkey@0"
I20260925 05:05:57.937533 20233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:57.949523 20233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:57.951105 20233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:57.951200 20233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:57.951354 20233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:57.951570 20233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:57.951596 20233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.951614 20233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:57.951627 20233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:57.956683 20233 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:36923
I20260925 05:05:57.956759 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:36923 every 8 connection(s)
I20260925 05:05:57.956997 20233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:57.959736 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20233
I20260925 05:05:57.959833 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:57.959977 14468 external_mini_cluster.cc:1428] Setting key 1d72b1de7facd6e1c4c5a1d8f13aa370
Sep 25 05:05:57 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:57.965613 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:57.957624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:48467 (local address 127.14.33.62:45643)
0925 05:05:57.957677 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:57.957679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:57.957859 (+ 180us) server_negotiation.cc:407] Connection header received
0925 05:05:57.958580 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:57.958581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:57.958590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:57.958634 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:57.959090 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.959509 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.960749 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.960934 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.960975 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:57.962958 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:57.962959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:57.962959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:57.962965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:57.964256 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:57.964702 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:57.964702 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:57.964703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:57.964747 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:57.965005 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:57.965006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:57.965006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:57.965091 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:57.965196 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:57.965494 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 05:05:57.965554 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:05:57.965844 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:57.957557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45643 (local address 127.14.33.2:48467)
0925 05:05:57.957745 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:57.957773 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:57.958490 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:57.958676 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:57.958678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:57.958878 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:57.959028 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:57.959031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.959591 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.959592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:57.960645 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:57.960664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:57.961003 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:57.961004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:57.961167 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:57.961552 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:57.961561 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:57.962886 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:57.964333 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:57.964339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:57.964342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:57.964624 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:57.964813 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:57.964814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:57.964814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:57.964944 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:57.965197 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:57.965202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:57.965354 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 05:05:57.965464 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:05:57.965618 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260925 05:05:57.966285 20362 heartbeater.cc:347] Connected to a master server at 127.14.33.62:45643
I20260925 05:05:57.966367 20362 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:57.966542 20362 heartbeater.cc:511] Master 127.14.33.62:45643 requested a full tablet report, sending...
I20260925 05:05:57.966849 20037 ts_manager.cc:194] Registered new tserver with Master: a85567cbea964a23b3b2fd3f70c63361 (127.14.33.2:36923)
I20260925 05:05:57.967180 20037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:48467
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:05:57.983371 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45643
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:58.060883 20367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:58.061071 20367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:58.061092 20367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:58.062156 20367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:58.062197 20367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:58.062225 20367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:58.063596 20367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:58.063766 20367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:58.063953 20367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:58.068571 20375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:58.068562 20373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:58.068548 20372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:58.068756 20367 server_base.cc:1034] running on GCE node
I20260925 05:05:58.068960 20367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:58.069254 20367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:58.070407 20367 hybrid_clock.cc:648] HybridClock initialized: now 1790312758070397 us; error 27 us; skew 500 ppm
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.072677 20367 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:05:58.073100 20367 webserver.cc:460] Webserver started at http://127.14.33.3:36039/ using document root <none> and password file <none>
I20260925 05:05:58.073328 20367 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:58.073367 20367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:58.073451 20367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:58.074359 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "7e900bdc33774f07821e8c0371d2fd35"
format_stamp: "Formatted at 2026-09-25 05:05:58 on dist-test-slave-4dmq"
server_key: "6a5c717aa72843fd73b760f5a0937939"
server_key_iv: "2985069a8de10b764cc273880b543446"
server_key_version: "encryptionkey@0"
I20260925 05:05:58.074641 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "7e900bdc33774f07821e8c0371d2fd35"
format_stamp: "Formatted at 2026-09-25 05:05:58 on dist-test-slave-4dmq"
server_key: "6a5c717aa72843fd73b760f5a0937939"
server_key_iv: "2985069a8de10b764cc273880b543446"
server_key_version: "encryptionkey@0"
I20260925 05:05:58.075845 20367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:05:58.076468 20382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:58.076612 20367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:05:58.076694 20367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "7e900bdc33774f07821e8c0371d2fd35"
format_stamp: "Formatted at 2026-09-25 05:05:58 on dist-test-slave-4dmq"
server_key: "6a5c717aa72843fd73b760f5a0937939"
server_key_iv: "2985069a8de10b764cc273880b543446"
server_key_version: "encryptionkey@0"
I20260925 05:05:58.076743 20367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:58.089036 20367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:58.091179 20367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:58.091284 20367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:58.091444 20367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:58.091684 20367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:58.091712 20367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:58.091732 20367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:58.091744 20367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:58.097311 20367 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:40973
I20260925 05:05:58.097344 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:40973 every 8 connection(s)
I20260925 05:05:58.097647 20367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:05:58.106734 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.098343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:48363 (local address 127.14.33.62:45643)
0925 05:05:58.098413 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.098415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.098548 (+ 133us) server_negotiation.cc:407] Connection header received
0925 05:05:58.099415 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.099416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.099442 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.099507 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.100091 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.100685 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.101737 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.101925 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.101973 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.103855 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:58.103856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:58.103856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:58.103862 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:58.105335 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.105808 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.105809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.105809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.105864 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.106114 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.106115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.106115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.106224 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:58.106301 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:58.106570 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:05:58.106643 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:05:58.106907 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.098301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45643 (local address 127.14.33.3:48363)
0925 05:05:58.098437 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:58.098469 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:58.099329 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:58.099567 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:58.099570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:58.099860 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:58.100023 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:58.100027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.100813 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.100814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:58.101653 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:58.101656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.101993 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.101994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:58.102084 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.102479 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:58.102486 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:58.103776 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:58.105460 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:58.105465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:58.105467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:58.105731 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:58.105929 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:58.105929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:58.105930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:58.106044 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:58.106291 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:58.106297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:58.106444 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 05:05:58.106548 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:05:58.106695 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260925 05:05:58.107374 20496 heartbeater.cc:347] Connected to a master server at 127.14.33.62:45643
I20260925 05:05:58.107465 20496 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:58.107630 20496 heartbeater.cc:511] Master 127.14.33.62:45643 requested a full tablet report, sending...
I20260925 05:05:58.107689 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20367
I20260925 05:05:58.107787 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:58.107973 14468 external_mini_cluster.cc:1428] Setting key 40765b508d0269d7599d4adf8ab95313
I20260925 05:05:58.108449 20036 ts_manager.cc:194] Registered new tserver with Master: 7e900bdc33774f07821e8c0371d2fd35 (127.14.33.3:40973)
I20260925 05:05:58.108872 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:05:58.108994 20036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:48363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/test-admin.keytab.
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.124953 14468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 05:05:58.126631 20503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:58.125842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.14.33.1:38013)
0925 05:05:58.126042 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.126047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.126055 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:58.126077 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.126079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.126092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.126174 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.126528 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57198: BlockingRecv error: recv got EOF from 127.0.0.1:57198 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
W20260925 05:05:58.126787 14468 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.127699 14468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.128669 14468 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:58.136243 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.129220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.14.33.1:38013)
0925 05:05:58.129281 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.129283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.129388 (+ 105us) server_negotiation.cc:407] Connection header received
0925 05:05:58.129461 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.129462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.129471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.129525 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.129922 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.130605 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.131329 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.131631 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.131707 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.133650 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:58.133658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:58.133658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:58.133665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:58.135129 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.135469 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.135472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.135473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.135527 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.135710 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.135710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.135711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.135851 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:58.135987 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:58.136067 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:05:58.136138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 05:05:58.137951 20503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:58.137449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.14.33.1:38013)
0925 05:05:58.137502 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.137503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.137583 (+ 80us) server_negotiation.cc:407] Connection header received
0925 05:05:58.137619 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.137619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.137627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.137667 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.137893 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57208: BlockingRecv error: recv got EOF from 127.0.0.1:57208 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 05:05:58.138047 14468 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.138904 14468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.139775 14468 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:58.145977 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.140211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.14.33.1:38013)
0925 05:05:58.140242 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.140243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.140287 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:05:58.140332 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.140332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.140338 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.140383 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.140743 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.141322 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.141926 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.142112 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.142154 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.143901 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:58.143902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:58.143902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:58.143908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:58.145223 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.145489 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.145490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.145490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.145534 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.145672 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.145672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.145672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.145739 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:58.145799 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:58.145877 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:05:58.145928 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 05:05:58.147336 20503 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:05:58.146831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.14.33.1:38013)
0925 05:05:58.146879 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.146881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.146910 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:05:58.146977 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.146978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.146986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.147037 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.147291 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57236: BlockingRecv error: recv got EOF from 127.0.0.1:57236 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 05:05:58.147387 14468 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312729583056-14468-0/krb5kdc/krb5cc' of type FILE
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.148227 14468 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:58.149047 14468 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[19992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:05:58.155216 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:58.149554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.14.33.1:38013)
0925 05:05:58.149603 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:58.149605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:58.149635 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:05:58.149703 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:58.149703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:58.149711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:58.149761 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:58.150125 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.150544 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.151164 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:58.151341 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:58.151404 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:58.153065 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:58.153066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:58.153067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:58.153073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:58.154185 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.154498 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.154499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.154499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.154545 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:58.154752 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:58.154753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:58.154753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:58.154848 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:58.154936 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:58.155056 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:05:58.155114 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:05:58.155812 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20099
I20260925 05:05:58.160121 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20233
I20260925 05:05:58.164198 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20367
I20260925 05:05:58.168625 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20005
2026-09-25T05:05:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1738 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[20516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[20516](info): set up 2 sockets
Sep 25 05:05:58 dist-test-slave-4dmq krb5kdc[20516](info): commencing operation
krb5kdc: starting...
W20260925 05:05:59.134209 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:05:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab.
Entry for principal oryx/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab.
I20260925 05:05:59.235759 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:39543
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab
--principal=oryx/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:59.309088 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:59.309329 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:59.309362 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:59.310475 20529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:59.310534 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:59.310554 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:59.310573 20529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:59.310598 20529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:59.312493 20529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab
--principal=oryx/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:59.312742 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:59.312956 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:59.317350 20535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.317422 20529 server_base.cc:1034] running on GCE node
W20260925 05:05:59.317363 20534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.317361 20537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.317821 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:59.318105 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:59.319260 20529 hybrid_clock.cc:648] HybridClock initialized: now 1790312759319242 us; error 46 us; skew 500 ppm
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:59.324911 20529 init.cc:377] Logged in from keytab as oryx/127.14.33.62@KRBTEST.COM (short username oryx)
I20260925 05:05:59.325357 20529 webserver.cc:460] Webserver started at http://127.14.33.62:37743/ using document root <none> and password file <none>
I20260925 05:05:59.325531 20529 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:59.325565 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:59.325641 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:59.326500 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "eda2cc968b064952b2670248d6ffdfca"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "5ae6a3167b2c830442299178811ada04"
server_key_iv: "65d0dbf6d29bcddc690834714882b447"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.326763 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "eda2cc968b064952b2670248d6ffdfca"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "5ae6a3167b2c830442299178811ada04"
server_key_iv: "65d0dbf6d29bcddc690834714882b447"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.327845 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:05:59.328444 20544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.328580 20529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:59.328655 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "eda2cc968b064952b2670248d6ffdfca"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "5ae6a3167b2c830442299178811ada04"
server_key_iv: "65d0dbf6d29bcddc690834714882b447"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.328708 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:59.342352 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:59.343714 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:59.343799 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:59.346870 20529 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:39543
I20260925 05:05:59.346939 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:39543 every 8 connection(s)
I20260925 05:05:59.347214 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:05:59.348027 20597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:59.349730 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20529
I20260925 05:05:59.349802 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:05:59.349978 14468 external_mini_cluster.cc:1428] Setting key 70cc893c5106a92e6803bb52ab30f02e
I20260925 05:05:59.350204 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Bootstrap starting.
I20260925 05:05:59.350840 20597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:59.351135 20597 log.cc:824] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:59.351800 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: No bootstrap required, opened a new log
I20260925 05:05:59.353479 20597 raft_consensus.cc:348] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } }
I20260925 05:05:59.353615 20597 raft_consensus.cc:374] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:59.353658 20597 raft_consensus.cc:729] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda2cc968b064952b2670248d6ffdfca, State: Initialized, Role: FOLLOWER
I20260925 05:05:59.353785 20597 consensus_queue.cc:260] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } }
I20260925 05:05:59.353893 20597 raft_consensus.cc:388] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:05:59.353933 20597 raft_consensus.cc:482] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:05:59.353977 20597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:05:59.354422 20597 raft_consensus.cc:504] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } }
I20260925 05:05:59.354527 20597 leader_election.cc:302] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eda2cc968b064952b2670248d6ffdfca; no voters:
I20260925 05:05:59.354660 20597 leader_election.cc:288] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:05:59.354699 20602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:05:59.354815 20602 raft_consensus.cc:686] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 LEADER]: Becoming Leader. State: Replica: eda2cc968b064952b2670248d6ffdfca, State: Running, Role: LEADER
I20260925 05:05:59.354888 20602 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } }
I20260925 05:05:59.354928 20597 sys_catalog.cc:564] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:59.355159 20602 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda2cc968b064952b2670248d6ffdfca. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } }
I20260925 05:05:59.355240 20602 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:05:59.355170 20603 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } }
I20260925 05:05:59.355409 20603 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
W20260925 05:05:59.356564 20618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:05:59.356614 20618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:05:59.356662 20607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:05:59.356839 20607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:05:59.358320 20607 catalog_manager.cc:1347] Generated new cluster ID: e10689a784b4439080aa2e998ce35e19
I20260925 05:05:59.358371 20607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:05:59.359747 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.350424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.14.33.62:39543)
0925 05:05:59.350585 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.350594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.350625 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:05:59.351204 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.351212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.351266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.351394 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.352002 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.352788 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.353524 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.354019 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.354115 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.356576 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.356580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.356580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.356591 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.358346 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.358682 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.358684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.358685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.358775 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.358971 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.358972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.358972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.359185 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.359339 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.359439 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.359559 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260925 05:05:59.381181 20607 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:05:59.381623 20607 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:05:59.391083 20607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Generated new TSK 0
I20260925 05:05:59.391273 20607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab.
Entry for principal oryx/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab.
I20260925 05:05:59.416662 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39543
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:59.490084 20623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:59.490252 20623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:59.490272 20623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:59.491308 20623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:59.491345 20623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:59.491393 20623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:05:59.492926 20623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:59.493188 20623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:59.493430 20623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:59.497970 20631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.497977 20629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.498113 20623 server_base.cc:1034] running on GCE node
W20260925 05:05:59.498256 20628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.498477 20623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:59.498746 20623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:59.499897 20623 hybrid_clock.cc:648] HybridClock initialized: now 1790312759499881 us; error 27 us; skew 500 ppm
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:59.502187 20623 init.cc:377] Logged in from keytab as oryx/127.14.33.1@KRBTEST.COM (short username oryx)
I20260925 05:05:59.502544 20623 webserver.cc:460] Webserver started at http://127.14.33.1:39369/ using document root <none> and password file <none>
I20260925 05:05:59.502739 20623 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:59.502799 20623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:59.502929 20623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:59.503841 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "2c1d9d241f924f2384681a943df80a18"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "76f1f4872d8ce5eb77a069b7eb9933c8"
server_key_iv: "c3445367cb6521ca060664e4eb5af7d5"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.504261 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "2c1d9d241f924f2384681a943df80a18"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "76f1f4872d8ce5eb77a069b7eb9933c8"
server_key_iv: "c3445367cb6521ca060664e4eb5af7d5"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.505726 20623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:05:59.506379 20638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.506560 20623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:59.506659 20623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "2c1d9d241f924f2384681a943df80a18"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "76f1f4872d8ce5eb77a069b7eb9933c8"
server_key_iv: "c3445367cb6521ca060664e4eb5af7d5"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.506739 20623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:59.513760 20623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:59.515816 20623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:59.515955 20623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:59.516139 20623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:59.516402 20623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:59.516453 20623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.516486 20623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:59.516511 20623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.521973 20623 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39143
I20260925 05:05:59.522024 20751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39143 every 8 connection(s)
I20260925 05:05:59.522334 20623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:05:59.530275 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.523012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:39473 (local address 127.14.33.62:39543)
0925 05:05:59.523073 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.523075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.523236 (+ 161us) server_negotiation.cc:407] Connection header received
0925 05:05:59.523656 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.523657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.523665 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.523714 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.524528 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.524989 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.525887 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.526100 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.526148 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.527943 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.527944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.527944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.527950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.529292 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.529596 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.529597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.529597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.529642 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.529828 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.529829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.529829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.529916 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.529961 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.530149 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.530217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:05:59.530469 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.522998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:39473)
0925 05:05:59.523128 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.523161 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.523580 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.523774 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.523777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.524293 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.524457 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.524461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.525075 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.525076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.525826 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.525828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.526164 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.526165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.526262 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.526614 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.526621 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.527865 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.529354 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.529360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.529361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.529538 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.529692 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.529693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.529693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.529780 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.529963 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.529967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.530056 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.530123 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.530217 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20260925 05:05:59.530512 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20623
I20260925 05:05:59.530604 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:05:59.530766 14468 external_mini_cluster.cc:1428] Setting key 5cdbdead07a6cfc15d8a439dc1b319e2
I20260925 05:05:59.531073 20752 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39543
I20260925 05:05:59.531164 20752 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:59.531316 20752 heartbeater.cc:511] Master 127.14.33.62:39543 requested a full tablet report, sending...
I20260925 05:05:59.531734 20561 ts_manager.cc:194] Registered new tserver with Master: 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143)
I20260925 05:05:59.532308 20561 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:39473
WARNING: no policy specified for oryx/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab.
Entry for principal oryx/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab.
I20260925 05:05:59.553571 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39543
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:59.628371 20757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:59.628535 20757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:59.628554 20757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:59.629518 20757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:59.629547 20757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:59.629570 20757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:05:59.630795 20757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:59.630940 20757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:59.631085 20757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:59.635459 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.635471 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.635466 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.635506 20757 server_base.cc:1034] running on GCE node
I20260925 05:05:59.635908 20757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:59.636149 20757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:59.637292 20757 hybrid_clock.cc:648] HybridClock initialized: now 1790312759637284 us; error 38 us; skew 500 ppm
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:59.639348 20757 init.cc:377] Logged in from keytab as oryx/127.14.33.2@KRBTEST.COM (short username oryx)
I20260925 05:05:59.639786 20757 webserver.cc:460] Webserver started at http://127.14.33.2:33737/ using document root <none> and password file <none>
I20260925 05:05:59.639985 20757 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:59.640044 20757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:59.640187 20757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:59.641017 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "b9acf3f3d9af4931ba9623540c32ddfa"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "f3ce0a81bffd0bc2d8b0f693d2910b75"
server_key_iv: "b4084189b2663d5907799609d32f56f7"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.641368 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "b9acf3f3d9af4931ba9623540c32ddfa"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "f3ce0a81bffd0bc2d8b0f693d2910b75"
server_key_iv: "b4084189b2663d5907799609d32f56f7"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.642475 20757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:05:59.643136 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.643311 20757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:05:59.643412 20757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "b9acf3f3d9af4931ba9623540c32ddfa"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "f3ce0a81bffd0bc2d8b0f693d2910b75"
server_key_iv: "b4084189b2663d5907799609d32f56f7"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.643496 20757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:59.658114 20757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:59.659680 20757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:59.659818 20757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:59.659997 20757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:59.660249 20757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:59.660313 20757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.660349 20757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:59.660373 20757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.665750 20757 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:39747
I20260925 05:05:59.665798 20885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:39747 every 8 connection(s)
I20260925 05:05:59.666039 20757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:05:59.667867 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20757
I20260925 05:05:59.667963 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:05:59.668134 14468 external_mini_cluster.cc:1428] Setting key d9e420ab95d721e8f29adcb9f8bb215f
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:05:59.675477 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.666663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:54467 (local address 127.14.33.62:39543)
0925 05:05:59.666722 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.666724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.666902 (+ 178us) server_negotiation.cc:407] Connection header received
0925 05:05:59.667312 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.667313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.667322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.667371 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.668385 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.668999 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.670392 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.670681 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.670760 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.672979 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.672980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.672981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.672987 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.674215 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.674653 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.674654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.674654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.674697 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.674923 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.674924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.674924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.675012 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.675057 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.675365 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.675422 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:05:59.675738 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.666630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:54467)
0925 05:05:59.666790 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.666826 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.667241 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.667414 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.667417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.668051 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.668293 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.668298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.669082 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.669084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.670310 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.670313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.670760 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.670760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.670856 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.671298 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.671306 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.672898 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.674290 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.674296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.674298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.674586 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.674740 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.674740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.674741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.674863 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.675075 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.675080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.675233 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.675349 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.675492 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260925 05:05:59.676151 20886 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39543
I20260925 05:05:59.676241 20886 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:59.676380 20886 heartbeater.cc:511] Master 127.14.33.62:39543 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.33.3@KRBTEST.COM; defaulting to no policy
I20260925 05:05:59.676682 20561 ts_manager.cc:194] Registered new tserver with Master: b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747)
I20260925 05:05:59.677030 20561 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:54467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab.
Entry for principal oryx/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab.
I20260925 05:05:59.691860 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39543
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:59.766371 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:59.766562 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:59.766584 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:59.767705 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:59.767745 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:59.767778 20891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:05:59.769217 20891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:05:59.769428 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:59.769629 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:05:59.774163 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.774211 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.774281 20891 server_base.cc:1034] running on GCE node
W20260925 05:05:59.774300 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.774500 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:59.774716 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:59.775861 20891 hybrid_clock.cc:648] HybridClock initialized: now 1790312759775841 us; error 38 us; skew 500 ppm
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:59.778156 20891 init.cc:377] Logged in from keytab as oryx/127.14.33.3@KRBTEST.COM (short username oryx)
I20260925 05:05:59.778591 20891 webserver.cc:460] Webserver started at http://127.14.33.3:40235/ using document root <none> and password file <none>
I20260925 05:05:59.778798 20891 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:59.778858 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:59.778995 20891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:59.779927 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "294bef61bf664c5689c561d6e8bec574"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c81e1e6d529ba529c1938575804d927d"
server_key_iv: "ae791607a525e2aec7610b635d37cba5"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.780273 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "294bef61bf664c5689c561d6e8bec574"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c81e1e6d529ba529c1938575804d927d"
server_key_iv: "ae791607a525e2aec7610b635d37cba5"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.781451 20891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 05:05:59.782059 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.782218 20891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.782316 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "294bef61bf664c5689c561d6e8bec574"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c81e1e6d529ba529c1938575804d927d"
server_key_iv: "ae791607a525e2aec7610b635d37cba5"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.782380 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:59.803531 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:59.805645 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:59.805804 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:59.806011 20891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:05:59.806335 20891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:05:59.806397 20891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.806433 20891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:05:59.806461 20891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.812285 20891 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:43701
I20260925 05:05:59.812321 21019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:43701 every 8 connection(s)
I20260925 05:05:59.812597 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:05:59.816496 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 20891
I20260925 05:05:59.816713 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:05:59.816907 14468 external_mini_cluster.cc:1428] Setting key e234344778b18f03ebb9af5faa67b857
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:05:59.823889 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.813293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:58923 (local address 127.14.33.62:39543)
0925 05:05:59.813332 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.813334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.813525 (+ 191us) server_negotiation.cc:407] Connection header received
0925 05:05:59.813921 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.813922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.813930 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.813978 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.814735 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.815224 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.816076 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.816232 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.816282 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.818851 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.818853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.818853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.818861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.822161 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.822683 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.822685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.822685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.822762 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.823084 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.823085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.823086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.823262 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.823333 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.823695 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.823781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:05:59.824072 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.813292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:58923)
0925 05:05:59.813417 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.813446 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.813849 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.814022 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.814024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.814496 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.814667 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.814671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.815303 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.815304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.816008 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.816010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.816295 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.816295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.816384 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.816893 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.816900 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.818599 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.822240 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.822245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.822247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.822593 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.822818 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.822832 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.822832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.822973 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.823334 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.823340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.823529 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.823650 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.823834 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260925 05:05:59.824515 21020 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39543
I20260925 05:05:59.824595 21020 heartbeater.cc:464] Registering TS with master...
I20260925 05:05:59.824759 21020 heartbeater.cc:511] Master 127.14.33.62:39543 requested a full tablet report, sending...
I20260925 05:05:59.825148 20561 ts_manager.cc:194] Registered new tserver with Master: 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701)
I20260925 05:05:59.825627 20561 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:58923
I20260925 05:05:59.828637 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.61.keytab.
Entry for principal oryx/127.14.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.61.keytab.
I20260925 05:05:59.851176 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:33121
--webserver_interface=127.14.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.61.keytab
--principal=oryx/127.14.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:39543,127.14.33.61:33121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:05:59.925628 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:05:59.925791 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:05:59.925808 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:05:59.926828 21025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:05:59.926864 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:05:59.926877 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:05:59.926920 21025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:05:59.926944 21025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:05:59.928212 21025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:39543,127.14.33.61:33121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.61.keytab
--principal=oryx/127.14.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
W20260925 05:05:59.928326 21025 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.33.62:39543,127.14.33.61:33121'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 05:05:59.928452 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:05:59.928647 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:05:59.932899 21025 server_base.cc:1034] running on GCE node
W20260925 05:05:59.932894 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.933003 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:05:59.932895 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:05:59.933272 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:05:59.933482 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:05:59.934608 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790312759934590 us; error 36 us; skew 500 ppm
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:05:59.936719 21025 init.cc:377] Logged in from keytab as oryx/127.14.33.61@KRBTEST.COM (short username oryx)
I20260925 05:05:59.937067 21025 webserver.cc:460] Webserver started at http://127.14.33.61:44731/ using document root <none> and password file <none>
I20260925 05:05:59.937288 21025 fs_manager.cc:359] Metadata directory not provided
I20260925 05:05:59.937330 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:05:59.937408 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:05:59.938253 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/instance:
uuid: "f25eb31273c0475fa6f42d55fa592f73"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c7b312f14ee7ae935afa93e40bf35925"
server_key_iv: "a96efdb95d676e322b0a08709f6842c1"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.938512 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal/instance:
uuid: "f25eb31273c0475fa6f42d55fa592f73"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c7b312f14ee7ae935afa93e40bf35925"
server_key_iv: "a96efdb95d676e322b0a08709f6842c1"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.939643 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:05:59.940227 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.940361 21025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:05:59.940434 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
uuid: "f25eb31273c0475fa6f42d55fa592f73"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c7b312f14ee7ae935afa93e40bf35925"
server_key_iv: "a96efdb95d676e322b0a08709f6842c1"
server_key_version: "encryptionkey@0"
I20260925 05:05:59.940478 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:05:59.956684 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:05:59.958258 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:05:59.958351 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:05:59.961772 21025 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.61:33121
I20260925 05:05:59.961797 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.61:33121 every 8 connection(s)
I20260925 05:05:59.962092 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
I20260925 05:05:59.962926 21093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:05:59.964316 21093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:05:59.964731 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 21025
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.61@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
I20260925 05:05:59.973348 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.965059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.14.33.62:39543)
0925 05:05:59.965160 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.965163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.965386 (+ 223us) server_negotiation.cc:407] Connection header received
0925 05:05:59.965954 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.965955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.965965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.966048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.966586 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.967088 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.968107 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.968405 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.968458 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.970591 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.970592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.970592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.970598 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.971883 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.972313 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.972315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.972315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.972373 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.972603 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.972604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.972605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.972790 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.972862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.973167 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.973244 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:05:59.973551 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.965018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41164)
0925 05:05:59.965244 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.965285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.965841 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.966100 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.966102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.966360 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.966495 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.966499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.967180 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.967189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.968033 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.968035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.968476 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.968477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.968598 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.969078 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.969084 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.970511 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.971964 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.971970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.971972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.972226 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.972439 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.972440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.972440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.972537 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.972872 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.972877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.972976 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.973102 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.973265 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20260925 05:05:59.974094 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.965250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.14.33.61:33121)
0925 05:05:59.965389 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.965392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.965439 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:05:59.965844 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.965855 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.965879 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.965959 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.966650 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.967610 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.968399 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.968811 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.968869 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.971113 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.971118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.971118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.971124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.972679 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.973028 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.973030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.973030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.973087 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.973310 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.973310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.973311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.973452 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.973587 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.973924 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.974005 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 05:05:59.974114 21093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.61@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
I20260925 05:05:59.979327 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.974358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49868)
0925 05:05:59.974383 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.974392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.974424 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.974602 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.974602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.974710 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.974733 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.974734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.975249 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.975249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.975676 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.975677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.975965 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.975965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.975997 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.976273 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.976276 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.977261 (+ 985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.978675 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.978675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.978676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.978809 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.978975 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.978976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.978976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.979019 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.979153 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.979154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.979179 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.979219 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.979280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260925 05:05:59.979382 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.974387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.14.33.61:33121)
0925 05:05:59.974443 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.974445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.974453 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:05:59.974468 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.974469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.974478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.974530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.974778 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.975179 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.975747 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.975911 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.975954 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.977345 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.977346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.977346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.977352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.978614 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.978885 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.978886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.978886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.978926 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.979052 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.979053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.979053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.979122 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.979159 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.979232 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.979319 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":41}
I20260925 05:05:59.980423 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Bootstrap starting.
I20260925 05:05:59.981169 21093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Neither blocks nor log segments found. Creating new log.
I20260925 05:05:59.981441 21093 log.cc:824] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Log is configured to *not* fsync() on all Append() calls
I20260925 05:05:59.982035 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: No bootstrap required, opened a new log
I20260925 05:05:59.983242 21093 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } }
I20260925 05:05:59.983368 21093 raft_consensus.cc:374] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:05:59.983410 21093 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25eb31273c0475fa6f42d55fa592f73, State: Initialized, Role: FOLLOWER
I20260925 05:05:59.983521 21093 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } }
I20260925 05:05:59.983731 21099 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } } }
I20260925 05:05:59.983773 21093 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:05:59.983801 21099 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:05:59.985169 21110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:05:59.985220 21110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:05:59.986605 21025 master_runner.cc:428] Detected that this master f25eb31273c0475fa6f42d55fa592f73 is joining an existing cluster
I20260925 05:05:59.986667 21025 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.33.61:33121
I20260925 05:05:59.993547 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.987649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.14.33.62:39543)
0925 05:05:59.987701 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.987703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.987978 (+ 275us) server_negotiation.cc:407] Connection header received
0925 05:05:59.988007 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.988008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.988018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.988073 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.988423 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.988920 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.989655 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.989840 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.989915 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.990810 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.990811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.990811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.990818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.992250 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.992733 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.992734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.992735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.992800 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.992986 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.992987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.992988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.993088 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.993213 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.993407 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.993468 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:05:59.993546 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.987615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41180)
0925 05:05:59.987887 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.987908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.987966 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.988087 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.988088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.988270 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.988345 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.988347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.988993 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.988994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.989573 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.989574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.989927 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.989928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.989980 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.990372 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.990377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.990702 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.992344 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.992344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.992345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.992655 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.992859 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.992859 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.992860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.992922 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.993237 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.993237 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.993285 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.993364 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.993454 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":72,"threads_started":1}
I20260925 05:05:59.994006 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49878)
0925 05:05:59.987932 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.987939 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.988032 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.988159 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.988164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.988318 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.988364 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.988365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.989181 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.989182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.989950 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.989953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.990367 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.990368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.990426 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.990769 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.990773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:05:59.991091 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:05:59.992873 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.992873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.992873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.993068 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.993296 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:05:59.993297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:05:59.993297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:05:59.993361 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:05:59.993674 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:05:59.993675 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:05:59.993718 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 05:05:59.993783 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 05:05:59.993875 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260925 05:05:59.994158 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.987787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.14.33.61:33121)
0925 05:05:59.987816 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.987819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.988003 (+ 184us) server_negotiation.cc:407] Connection header received
0925 05:05:59.988091 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.988091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.988098 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.988119 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.988458 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.989072 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.990034 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.990275 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.990310 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.991196 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:05:59.991197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:05:59.991197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:05:59.991202 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:05:59.992796 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.993181 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.993182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.993183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.993230 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:05:59.993432 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:05:59.993433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:05:59.993433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:05:59.993537 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:05:59.993570 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 05:05:59.993825 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 05:05:59.993929 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":13}
Sep 25 05:05:59 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.62@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
I20260925 05:06:00.002781 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.995972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.14.33.61:33121)
0925 05:05:59.996002 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 05:05:59.996003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:05:59.996186 (+ 183us) server_negotiation.cc:407] Connection header received
0925 05:05:59.996264 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:05:59.996265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:05:59.996271 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:05:59.996312 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:05:59.996743 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.997199 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.998285 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.998449 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.998495 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.000330 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.000331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.000332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.000338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.001600 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.002006 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.002007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.002008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.002056 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.002301 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.002302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.002302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.002397 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.002468 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.002639 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.002704 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:06:00.002853 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:05:59.995925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49892)
0925 05:05:59.996079 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:05:59.996116 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:05:59.996195 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:05:59.996318 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:05:59.996320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:05:59.996613 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:05:59.996661 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.996663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.997277 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.997279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.998180 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:05:59.998182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:05:59.998518 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:05:59.998520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:05:59.998597 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:05:59.998980 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:05:59.998984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.000251 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.001719 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.001726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.001728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.001930 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.002139 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.002140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.002140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.002224 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.002467 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.002475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.002522 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.002615 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.002720 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260925 05:06:00.003374 20561 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.33.61:33121: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73"
member_type: NON_VOTER
last_known_addr {
host: "127.14.33.61"
port: 33121
}
attrs {
promote: true
}
}
dest_uuid: "eda2cc968b064952b2670248d6ffdfca"
cas_config_opid_index: -1
I20260925 05:06:00.003702 20561 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } }
I20260925 05:06:00.004135 21120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } }
I20260925 05:06:00.004421 21067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:00.005028 21120 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.33.61:33121
I20260925 05:06:00.005239 21025 master.cc:537] Master@127.14.33.61:33121 shutting down...
I20260925 05:06:00.005285 21067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Refusing update from remote peer eda2cc968b064952b2670248d6ffdfca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 05:06:00.005268 21119 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } }
I20260925 05:06:00.005342 21119 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.005599 21119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 05:06:00.006294 20548 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 05:06:00.006798 20548 consensus_peers.cc:487] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca -> Peer f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61:33121): Couldn't send request to peer f25eb31273c0475fa6f42d55fa592f73. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 05:06:00.007949 21025 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:00.008037 21025 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:00.008085 21025 tablet_replica.cc:330] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: stopping tablet replica
I20260925 05:06:00.020349 21025 master.cc:552] Master@127.14.33.61:33121 shutdown complete.
I20260925 05:06:00.020408 21025 master_runner.cc:305] Clearing existing system tablet
I20260925 05:06:00.020524 21025 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.020579 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.021157 21025 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:00.021641 21124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.021834 21025 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.021896 21025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.021926 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
uuid: "f25eb31273c0475fa6f42d55fa592f73"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c7b312f14ee7ae935afa93e40bf35925"
server_key_iv: "a96efdb95d676e322b0a08709f6842c1"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.022464 21025 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:00.023648 21025 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:06:00.023723 21025 log.cc:1196] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 05:06:00.023917 21025 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Deleting consensus metadata
I20260925 05:06:00.024042 21025 master_runner.cc:315] Copying system tablet from 127.14.33.62:39543
I20260925 05:06:00.024595 21025 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: tablet copy: Beginning tablet copy session from remote peer at address 127.14.33.62:39543
I20260925 05:06:00.030435 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.024883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.14.33.62:39543)
0925 05:06:00.024921 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.024923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.025060 (+ 137us) server_negotiation.cc:407] Connection header received
0925 05:06:00.025161 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.025162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.025174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.025237 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.025568 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.026014 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.027022 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.027186 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.027228 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.028126 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.028127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.028127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.028132 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.029489 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.029839 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.029839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.029840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.029897 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.030065 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.030066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.030066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.030147 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.030190 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.030335 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.030381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:06:00.030505 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.024836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41182)
0925 05:06:00.024984 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.024994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.025055 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.025256 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.025257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.025443 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.025495 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.025496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.026085 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.026086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.026942 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.026944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.027248 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.027249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.027312 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.027680 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.027691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.028043 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.029560 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.029561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.029562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.029773 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.029953 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.029954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.029954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.030011 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.030201 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.030202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.030240 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.030323 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.030411 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260925 05:06:00.031096 20581 tablet_copy_service.cc:140] P eda2cc968b064952b2670248d6ffdfca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f25eb31273c0475fa6f42d55fa592f73 ({username='oryx', principal='oryx/127.14.33.61@KRBTEST.COM'} at 127.0.0.1:41182)
I20260925 05:06:00.031201 20581 tablet_copy_service.cc:161] P eda2cc968b064952b2670248d6ffdfca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f25eb31273c0475fa6f42d55fa592f73 at {username='oryx', principal='oryx/127.14.33.61@KRBTEST.COM'} at 127.0.0.1:41182: session id = f25eb31273c0475fa6f42d55fa592f73-00000000000000000000000000000000
I20260925 05:06:00.031960 20581 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:06:00.032562 21025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:06:00.033624 21025 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: tablet copy: Starting download of 0 data blocks...
I20260925 05:06:00.033761 21025 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: tablet copy: Starting download of 1 WAL segments...
I20260925 05:06:00.034482 21025 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:06:00.035259 20581 tablet_copy_service.cc:342] P eda2cc968b064952b2670248d6ffdfca: Request end of tablet copy session f25eb31273c0475fa6f42d55fa592f73-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.33.61@KRBTEST.COM'} at 127.0.0.1:41182
I20260925 05:06:00.035308 20581 tablet_copy_service.cc:433] P eda2cc968b064952b2670248d6ffdfca: ending tablet copy session f25eb31273c0475fa6f42d55fa592f73-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f25eb31273c0475fa6f42d55fa592f73
I20260925 05:06:00.037894 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.039040 21131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.039022 21132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.039258 21025 server_base.cc:1034] running on GCE node
W20260925 05:06:00.039286 21134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.039412 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.039544 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.040668 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790312760040661 us; error 22 us; skew 500 ppm
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.041724 21025 init.cc:377] Logged in from keytab as oryx/127.14.33.61@KRBTEST.COM (short username oryx)
I20260925 05:06:00.041985 21025 webserver.cc:460] Webserver started at http://127.14.33.61:35261/ using document root <none> and password file <none>
I20260925 05:06:00.042098 21025 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.042140 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.042678 21025 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:00.043143 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.043257 21025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.043303 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
uuid: "f25eb31273c0475fa6f42d55fa592f73"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c7b312f14ee7ae935afa93e40bf35925"
server_key_iv: "a96efdb95d676e322b0a08709f6842c1"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.043351 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.054121 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.055547 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.058473 21025 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.61:33121
I20260925 05:06:00.058619 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.61:33121 every 8 connection(s)
I20260925 05:06:00.058810 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
I20260925 05:06:00.059427 21194 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:06:00.059811 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Bootstrap starting.
I20260925 05:06:00.061851 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:06:00.062062 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: Bootstrap complete.
I20260925 05:06:00.062233 21194 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } }
I20260925 05:06:00.062296 21194 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f25eb31273c0475fa6f42d55fa592f73, State: Initialized, Role: LEARNER
I20260925 05:06:00.062320 21194 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } }
I20260925 05:06:00.062469 21194 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:00.062477 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } }
I20260925 05:06:00.062620 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:06:00.063840 21208 catalog_manager.cc:1259] Loaded cluster ID: e10689a784b4439080aa2e998ce35e19
I20260925 05:06:00.063927 21208 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: loading cluster ID for follower catalog manager: success
I20260925 05:06:00.064844 21208 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: acquiring CA information for follower catalog manager: success
I20260925 05:06:00.065100 21208 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:06:00.081477 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.075530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.14.33.61:33121)
0925 05:06:00.075650 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.075653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.075668 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:06:00.075686 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.075686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.075694 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.075760 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.076274 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.076878 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.077616 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.077795 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.077870 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.079049 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.079050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.079050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.079056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.080347 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.080679 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.080680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.080680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.080739 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.080933 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.080934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.080934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.081039 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.081165 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.081283 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.081357 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for oryx/127.14.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.60.keytab.
Entry for principal oryx/127.14.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.60.keytab.
I20260925 05:06:00.104041 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:37515
--webserver_interface=127.14.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.60.keytab
--principal=oryx/127.14.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:00.180133 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:00.180312 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:00.180332 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:00.181507 21212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:00.181545 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:00.181558 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:00.181603 21212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:00.181628 21212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:00.183044 21212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.60.keytab
--principal=oryx/127.14.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:37515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:00.183297 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:00.183526 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.188891 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.188926 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.189024 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.189152 21212 server_base.cc:1034] running on GCE node
I20260925 05:06:00.189424 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.189711 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.190913 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790312760190892 us; error 34 us; skew 500 ppm
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.193464 21212 init.cc:377] Logged in from keytab as oryx/127.14.33.60@KRBTEST.COM (short username oryx)
I20260925 05:06:00.193873 21212 webserver.cc:460] Webserver started at http://127.14.33.60:45773/ using document root <none> and password file <none>
I20260925 05:06:00.194096 21212 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.194164 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.194301 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:00.195302 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/instance:
uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-4dmq"
server_key: "add3050b0e707e266a47ef4b93034955"
server_key_iv: "14638cfda58738fb34ad73a874161d19"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.195647 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal/instance:
uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-4dmq"
server_key: "add3050b0e707e266a47ef4b93034955"
server_key_iv: "14638cfda58738fb34ad73a874161d19"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.196864 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:06:00.197625 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.197788 21212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.197876 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-4dmq"
server_key: "add3050b0e707e266a47ef4b93034955"
server_key_iv: "14638cfda58738fb34ad73a874161d19"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.197957 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.208932 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.211001 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:00.211100 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.214630 21212 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.60:37515
I20260925 05:06:00.214699 21279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.60:37515 every 8 connection(s)
I20260925 05:06:00.214991 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
I20260925 05:06:00.215890 21280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:00.217357 21280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:06:00.218073 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 21212
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.60@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.33.60@KRBTEST.COM
I20260925 05:06:00.226610 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.218076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.14.33.62:39543)
0925 05:06:00.218160 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.218162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.218348 (+ 186us) server_negotiation.cc:407] Connection header received
0925 05:06:00.218944 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.218945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.218959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.219032 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.220002 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.220469 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.221409 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.221594 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.221663 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.223761 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.223762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.223762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.223768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.225328 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.225731 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.225732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.225732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.225779 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.226021 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.226021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.226021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.226107 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.226181 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.226450 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.226524 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:06:00.226843 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.217983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41190)
0925 05:06:00.218162 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.218244 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.218841 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.219628 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.219630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.219837 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.219913 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.219917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.220588 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.220589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.221325 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.221327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.221700 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.221702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.221763 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.222262 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.222269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.223683 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.225397 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.225403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.225405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.225660 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.225839 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.225840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.225840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.225954 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.226198 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.226203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.226320 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.226412 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.226537 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260925 05:06:00.227262 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.218594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.14.33.60:37515)
0925 05:06:00.218762 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.218767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.218798 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:06:00.218873 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.218880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.218901 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.219009 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.219624 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.220480 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.221230 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.221522 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.221630 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.224385 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.224392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.224392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.224398 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.226096 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.226411 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.226414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.226414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.226470 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.226656 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.226657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.226658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.226804 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.227061 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.227101 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.227169 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260925 05:06:00.227347 21280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.60@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
I20260925 05:06:00.232620 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.227598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.14.33.61:33121)
0925 05:06:00.227638 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.227640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.227647 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:00.227669 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.227670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.227690 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.227740 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.227955 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.228362 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.228907 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.229072 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.229159 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.230544 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.230545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.230546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.230553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.231910 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.232153 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.232154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.232154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.232193 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.232318 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.232319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.232319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.232393 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.232431 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.232517 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.232567 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:06:00.232755 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.227523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49906)
0925 05:06:00.227569 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.227578 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.227614 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.227764 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.227765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.227883 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.227905 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.227906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.228415 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.228416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.228844 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.228845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.229163 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.229164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.229211 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.229483 (+ 272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.229486 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.230454 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.231967 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.231968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.231968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.232098 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.232238 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.232239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.232239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.232285 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.232437 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.232437 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.232464 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.232504 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.232604 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:06:00.233186 21280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.60@KRBTEST.COM for oryx/127.14.33.60@KRBTEST.COM
I20260925 05:06:00.238193 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.233409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40146)
0925 05:06:00.233452 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.233460 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.233496 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.233592 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.233593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.233681 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.233702 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.233703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.234203 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.234203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.234636 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.234637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.234980 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.234980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.235011 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.235293 (+ 282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.235297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.236176 (+ 879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.237572 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.237572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.237573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.237701 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.237835 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.237835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.237835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.237877 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.238021 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.238021 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.238047 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.238090 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.238147 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 05:06:00.238276 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.233432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.14.33.60:37515)
0925 05:06:00.233482 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.233483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.233491 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:00.233551 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.233551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.233558 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.233590 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.233736 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.234138 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.234697 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.234921 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.234968 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.236242 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.236243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.236243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.236249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.237512 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.237751 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.237752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.237752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.237788 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.237910 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.237911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.237911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.237977 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.238015 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.238101 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.238195 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":35}
I20260925 05:06:00.239248 21280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Bootstrap starting.
I20260925 05:06:00.240029 21280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:00.240293 21280 log.cc:824] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:00.240922 21280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: No bootstrap required, opened a new log
I20260925 05:06:00.242164 21280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } }
I20260925 05:06:00.242350 21280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:00.242395 21280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fe7dc2d89e41e0ba921f3524f590a7, State: Initialized, Role: FOLLOWER
I20260925 05:06:00.242527 21280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } }
I20260925 05:06:00.242740 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } } }
I20260925 05:06:00.242796 21280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:00.242841 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:06:00.244395 21296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:00.244449 21296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:06:00.246398 21212 master_runner.cc:428] Detected that this master 19fe7dc2d89e41e0ba921f3524f590a7 is joining an existing cluster
I20260925 05:06:00.246438 21212 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.33.60:37515
I20260925 05:06:00.253543 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.247377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41202)
0925 05:06:00.247544 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.247553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.247656 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.247847 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.247847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.248042 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.248113 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.248114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.248908 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.248908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.249785 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.249787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.250251 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.250333 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.250803 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.250806 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.251078 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.252656 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.252657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.252657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.252827 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.253030 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.253031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.253031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.253075 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.253333 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.253334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.253375 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.253428 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.253488 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260925 05:06:00.253621 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.247397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.14.33.62:39543)
0925 05:06:00.247432 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.247435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.247629 (+ 194us) server_negotiation.cc:407] Connection header received
0925 05:06:00.247737 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.247738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.247748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.247789 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.248164 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.248832 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.249863 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.250145 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250197 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.251172 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.251174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.251174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.251181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.252575 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.252896 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.252898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.252898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.252956 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.253150 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.253151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.253152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.253270 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.253337 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.253471 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.253538 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.62@KRBTEST.COM for oryx/127.14.33.60@KRBTEST.COM
I20260925 05:06:00.269054 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.247521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49908)
0925 05:06:00.247636 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.247645 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.247789 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.247939 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.247939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.248325 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.248372 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.248374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.249164 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.249164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.249900 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.249901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.250390 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.250450 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.265895 (+ 15445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.265900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.266328 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.267960 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.267961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.267962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.268250 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.268438 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268438 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.268439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.268490 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.268752 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.268753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.268792 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.268861 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.268950 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260925 05:06:00.269080 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.255563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40156)
0925 05:06:00.255613 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.255624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.255667 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.255950 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.255951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.256106 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.256148 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.256150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.256736 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.256738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.257545 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.257548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.258122 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.258123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.258191 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.258516 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.258520 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.259988 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.268017 (+ 8029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.268019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.268242 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.268451 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.268452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.268511 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.268784 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.268785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.268831 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.268882 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.268983 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 05:06:00.269339 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.247450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.14.33.61:33121)
0925 05:06:00.247485 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.247486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.247733 (+ 247us) server_negotiation.cc:407] Connection header received
0925 05:06:00.247854 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.247854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.247863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.247889 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.248449 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.249059 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.249990 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.250276 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250315 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.266415 (+ 16100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.266416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.266416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.266422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.267871 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268325 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.268326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.268326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.268375 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268550 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.268551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.268551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.268674 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.268728 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.269170 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.269242 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 05:06:00.269539 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.247712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.14.33.60:37515)
0925 05:06:00.247771 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.247773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.248094 (+ 321us) server_negotiation.cc:407] Connection header received
0925 05:06:00.248119 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.248120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.248128 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.248179 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.248522 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.249161 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250120 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.250397 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250430 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.266121 (+ 15691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.266122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.266122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.266127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.268440 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268779 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.268779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.268780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.268822 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268988 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.268989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.268989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.269082 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.269119 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.269373 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.269450 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:06:00.269944 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.247705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40150)
0925 05:06:00.248022 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.248031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.248084 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.248209 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.248209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.248397 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.248447 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.248448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.249238 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.249239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.250034 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.250036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.250505 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.250505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.250558 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.265601 (+ 15043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.265605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.266038 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.268526 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.268527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.268709 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.268885 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.268886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.268928 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.269188 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.269189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.269240 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.269615 (+ 375us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.269684 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":78,"threads_started":1}
I20260925 05:06:00.270103 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.255676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.14.33.60:37515)
0925 05:06:00.255820 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.255822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.255828 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:06:00.255864 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.255864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.255872 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.255921 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.256216 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.256663 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.257616 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.258020 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.258085 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.260085 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.260087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.260087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.260092 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.267922 (+ 7830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268326 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.268327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.268327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.268380 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.268581 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.268582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.268582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.268706 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.269892 (+ 1186us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.269936 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.270003 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260925 05:06:00.270443 20561 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.33.60:37515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
member_type: NON_VOTER
last_known_addr {
host: "127.14.33.60"
port: 37515
}
attrs {
promote: true
}
}
dest_uuid: "eda2cc968b064952b2670248d6ffdfca"
cas_config_opid_index: 5
I20260925 05:06:00.270673 20561 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } }
I20260925 05:06:00.271011 21120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.271279 21254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:00.271852 21120 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.33.60:37515
I20260925 05:06:00.271894 21254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Refusing update from remote peer eda2cc968b064952b2670248d6ffdfca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:06:00.271914 21120 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.271971 21120 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.272141 21120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:00.272238 21212 master.cc:537] Master@127.14.33.60:37515 shutting down...
W20260925 05:06:00.272786 20548 consensus_peers.cc:487] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca -> Peer 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60:37515): Couldn't send request to peer 19fe7dc2d89e41e0ba921f3524f590a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 05:06:00.275050 21212 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:00.275131 21212 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:00.275185 21212 tablet_replica.cc:330] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: stopping tablet replica
I20260925 05:06:00.275530 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.271039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49914)
0925 05:06:00.271081 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.271090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.271127 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.272413 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.272414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.272416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.272449 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.272451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.273334 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.273335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.274469 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.274472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.275288 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.275289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.275352 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.275373 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.275422 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.275456 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 05:06:00.275708 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.272085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.14.33.61:33121)
0925 05:06:00.272243 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.272244 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.272251 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:00.272268 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.272268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.272289 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.272361 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.272514 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.273247 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.274562 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.275115 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.275184 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.275256 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.275614 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.275643 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260925 05:06:00.275873 21168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 LEARNER]: Refusing update from remote peer eda2cc968b064952b2670248d6ffdfca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 05:06:00.276068 21120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 05:06:00.276575 21168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.277632 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.277732 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:06:00.277796 21120 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: attempting to promote NON_VOTER f25eb31273c0475fa6f42d55fa592f73 to VOTER
I20260925 05:06:00.277820 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda2cc968b064952b2670248d6ffdfca. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: NON_VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: true } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.277931 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:06:00.277968 21120 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } }
I20260925 05:06:00.278403 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.278315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40164)
0925 05:06:00.278350 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.278384 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:37515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 05:06:00.278455 21168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 LEARNER]: Refusing update from remote peer eda2cc968b064952b2670248d6ffdfca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 05:06:00.278777 21119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 05:06:00.278841 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.278607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40166)
0925 05:06:00.278728 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.278765 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:37515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
W20260925 05:06:00.278921 20548 consensus_peers.cc:487] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca -> Peer 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60:37515): Couldn't send request to peer 19fe7dc2d89e41e0ba921f3524f590a7. Status: Network error: Client connection negotiation failed: client connection to 127.14.33.60:37515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:06:00.279422 21308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.279587 21168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.280268 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.280311 21119 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.280350 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.280362 21119 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.280566 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.280660 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:00.287902 21212 master.cc:552] Master@127.14.33.60:37515 shutdown complete.
I20260925 05:06:00.287969 21212 master_runner.cc:305] Clearing existing system tablet
I20260925 05:06:00.288097 21212 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.288152 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.288815 21212 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:00.289314 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.289430 21212 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.289484 21212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.289505 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-4dmq"
server_key: "add3050b0e707e266a47ef4b93034955"
server_key_iv: "14638cfda58738fb34ad73a874161d19"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.290071 21212 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:00.291291 21212 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 05:06:00.291348 21212 log.cc:1196] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 05:06:00.291541 21212 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Deleting consensus metadata
I20260925 05:06:00.291661 21212 master_runner.cc:315] Copying system tablet from 127.14.33.62:39543
I20260925 05:06:00.292253 21212 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.33.62:39543
I20260925 05:06:00.298187 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.292617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.14.33.62:39543)
0925 05:06:00.292665 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.292668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.292774 (+ 106us) server_negotiation.cc:407] Connection header received
0925 05:06:00.292852 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.292852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.292862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.292918 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.293281 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.293769 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.294492 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.294685 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.294762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.295692 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.295693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.295694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.295701 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.297086 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.297459 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.297460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.297460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.297530 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.297722 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.297723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.297724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.297829 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.297897 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.298031 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.298099 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:06:00.298208 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.292562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41206)
0925 05:06:00.292679 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.292699 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.292790 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.292922 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.292923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.293101 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.293199 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.293201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.293848 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.293850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.294401 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.294403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.294765 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.294766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.294829 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.295315 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.295320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.295601 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.297220 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.297221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.297221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.297389 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.297590 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.297591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.297591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.297643 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.297889 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.297890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.297931 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.298006 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.298100 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":65,"threads_started":1}
I20260925 05:06:00.298388 20581 tablet_copy_service.cc:140] P eda2cc968b064952b2670248d6ffdfca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 19fe7dc2d89e41e0ba921f3524f590a7 ({username='oryx', principal='oryx/127.14.33.60@KRBTEST.COM'} at 127.0.0.1:41206)
I20260925 05:06:00.298427 20581 tablet_copy_service.cc:161] P eda2cc968b064952b2670248d6ffdfca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 19fe7dc2d89e41e0ba921f3524f590a7 at {username='oryx', principal='oryx/127.14.33.60@KRBTEST.COM'} at 127.0.0.1:41206: session id = 19fe7dc2d89e41e0ba921f3524f590a7-00000000000000000000000000000000
I20260925 05:06:00.298863 20581 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 05:06:00.299321 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 05:06:00.300127 21212 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: tablet copy: Starting download of 0 data blocks...
I20260925 05:06:00.300283 21212 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: tablet copy: Starting download of 1 WAL segments...
I20260925 05:06:00.301014 21212 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 05:06:00.301705 20581 tablet_copy_service.cc:342] P eda2cc968b064952b2670248d6ffdfca: Request end of tablet copy session 19fe7dc2d89e41e0ba921f3524f590a7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.33.60@KRBTEST.COM'} at 127.0.0.1:41206
I20260925 05:06:00.301760 20581 tablet_copy_service.cc:433] P eda2cc968b064952b2670248d6ffdfca: ending tablet copy session 19fe7dc2d89e41e0ba921f3524f590a7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 19fe7dc2d89e41e0ba921f3524f590a7
I20260925 05:06:00.304067 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.305166 21318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.305208 21319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.305251 21321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.305374 21212 server_base.cc:1034] running on GCE node
I20260925 05:06:00.305462 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.305588 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.306710 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790312760306702 us; error 21 us; skew 500 ppm
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.307837 21212 init.cc:377] Logged in from keytab as oryx/127.14.33.60@KRBTEST.COM (short username oryx)
I20260925 05:06:00.308099 21212 webserver.cc:460] Webserver started at http://127.14.33.60:37699/ using document root <none> and password file <none>
I20260925 05:06:00.308231 21212 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.308274 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.308859 21212 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:00.309352 21328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.309456 21212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.309499 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
format_stamp: "Formatted at 2026-09-25 05:06:00 on dist-test-slave-4dmq"
server_key: "add3050b0e707e266a47ef4b93034955"
server_key_iv: "14638cfda58738fb34ad73a874161d19"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.309551 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.319378 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.320886 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.323917 21212 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.60:37515
I20260925 05:06:00.324077 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.60:37515 every 8 connection(s)
I20260925 05:06:00.324280 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
I20260925 05:06:00.324982 21381 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:06:00.325453 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Bootstrap starting.
I20260925 05:06:00.328145 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:06:00.328336 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Bootstrap complete.
I20260925 05:06:00.328526 21381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } }
I20260925 05:06:00.328598 21381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19fe7dc2d89e41e0ba921f3524f590a7, State: Initialized, Role: LEARNER
I20260925 05:06:00.328632 21381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } }
I20260925 05:06:00.328817 21381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:00.328840 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.329037 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:06:00.330219 21396 catalog_manager.cc:1259] Loaded cluster ID: e10689a784b4439080aa2e998ce35e19
I20260925 05:06:00.330400 21396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: loading cluster ID for follower catalog manager: success
I20260925 05:06:00.331832 21396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: acquiring CA information for follower catalog manager: success
I20260925 05:06:00.332140 21396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:06:00.335362 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.328509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.14.33.60:37515)
0925 05:06:00.328625 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.328627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.328634 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:00.328656 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.328657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.328666 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.329424 (+ 758us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.329454 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.330420 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.331353 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.331619 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.331670 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.332800 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.332801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.332802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.332808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.334300 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.334609 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.334610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.334611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.334680 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.334872 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.334873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.334873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.335000 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.335099 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.335176 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.335275 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260925 05:06:00.335805 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20623
I20260925 05:06:00.340252 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:39143
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:39543
--webserver_port=39369
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:00.418316 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:00.418495 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:00.418515 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:00.419552 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:00.419584 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:00.419606 21397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:00.420900 21397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:39143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=39369
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:00.421060 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:00.421288 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.425729 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.425719 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.425828 21397 server_base.cc:1034] running on GCE node
W20260925 05:06:00.425745 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.426111 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.426355 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.427505 21397 hybrid_clock.cc:648] HybridClock initialized: now 1790312760427501 us; error 37 us; skew 500 ppm
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.430042 21397 init.cc:377] Logged in from keytab as oryx/127.14.33.1@KRBTEST.COM (short username oryx)
I20260925 05:06:00.430469 21397 webserver.cc:460] Webserver started at http://127.14.33.1:39369/ using document root <none> and password file <none>
I20260925 05:06:00.430682 21397 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.430759 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.432111 21397 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:00.432611 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.432807 21397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:00.432893 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "2c1d9d241f924f2384681a943df80a18"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "76f1f4872d8ce5eb77a069b7eb9933c8"
server_key_iv: "c3445367cb6521ca060664e4eb5af7d5"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.433223 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.444108 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.445888 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:00.446022 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.446200 21397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:00.446478 21397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:00.446509 21397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.446528 21397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:00.446570 21397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.451752 21397 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39143
I20260925 05:06:00.451821 21525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39143 every 8 connection(s)
I20260925 05:06:00.452114 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:00.454341 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 21397
I20260925 05:06:00.454483 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20757
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.60@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:00.462541 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.452793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:40553 (local address 127.14.33.60:37515)
0925 05:06:00.452844 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.452846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.452983 (+ 137us) server_negotiation.cc:407] Connection header received
0925 05:06:00.453583 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.453584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.453597 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.453651 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.454289 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.454893 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.456042 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.456368 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.456427 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.459510 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.459511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.459511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.459519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.460995 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.461536 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.461537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.461538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.461600 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.461875 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.461876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.461877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.462026 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.462078 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.462374 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.462457 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:06:00.462591 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:39747
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:39543
--webserver_port=33737
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:00.462670 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.452744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.14.33.1:40553)
0925 05:06:00.452898 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.452914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.453500 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.453708 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.453712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.454040 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.454200 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.454203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.454999 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.455000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.455952 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.455954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.456486 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.456487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.456670 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.457741 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.457746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.459413 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.461099 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.461104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.461106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.461393 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.461670 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.461671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.461671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.461796 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.462108 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.462113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.462242 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.462337 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.462471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":60,"thread_start_us":86,"threads_started":1}
I20260925 05:06:00.462998 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.453198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:51785 (local address 127.14.33.61:33121)
0925 05:06:00.453255 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.453257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.453264 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:00.453605 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.453606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.453616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.453647 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.454304 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.454991 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.456234 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.456518 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.456576 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.459185 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.459187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.459187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.459206 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.461499 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.461873 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.461874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.461874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.461925 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.462124 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.462125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.462126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.462277 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.462359 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.462843 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.462915 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:06:00.463011 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.452972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.14.33.1:51785)
0925 05:06:00.453190 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.453199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.453500 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.453710 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.453713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.454118 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.454201 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.454208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.455078 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.455078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.456138 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.456141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.456626 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.456627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.456676 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.457321 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.457330 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.459086 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.461563 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.461564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.461564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.461753 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.462007 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.462007 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.462008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.462059 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.462653 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.462654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.462714 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.462815 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.462923 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260925 05:06:00.463980 21528 heartbeater.cc:347] Connected to a master server at 127.14.33.61:33121
I20260925 05:06:00.464008 21526 heartbeater.cc:347] Connected to a master server at 127.14.33.60:37515
I20260925 05:06:00.464069 21528 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.464068 21526 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.464252 21526 heartbeater.cc:511] Master 127.14.33.60:37515 requested a full tablet report, sending...
I20260925 05:06:00.464254 21528 heartbeater.cc:511] Master 127.14.33.61:33121 requested a full tablet report, sending...
I20260925 05:06:00.464710 21158 ts_manager.cc:194] Registered new tserver with Master: 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143)
I20260925 05:06:00.464710 21345 ts_manager.cc:194] Registered new tserver with Master: 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143)
I20260925 05:06:00.465169 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.452754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:36213 (local address 127.14.33.62:39543)
0925 05:06:00.453084 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.453086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.453094 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:00.453606 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.453607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.453618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.453647 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.456788 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.457560 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.458675 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.458998 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.459067 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.461202 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.461204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.461204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.461212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.463018 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.464384 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.464385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.464385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.464447 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.464629 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.464630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.464630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.464747 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.464786 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.464979 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.465056 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260925 05:06:00.465400 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.452721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:36213)
0925 05:06:00.452879 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.452913 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.453500 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.453715 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.453715 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.454069 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.456693 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.456695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.457653 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.457654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.458540 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.458542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.459163 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.459164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.459228 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.459644 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.459649 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.461075 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.463755 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.463756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.463756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.464296 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.464516 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.464516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.464517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.464568 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.464827 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.464828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.464873 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.465255 (+ 382us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.465344 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260925 05:06:00.465673 21527 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39543
I20260925 05:06:00.465713 21527 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.465783 21527 heartbeater.cc:511] Master 127.14.33.62:39543 requested a full tablet report, sending...
I20260925 05:06:00.466035 20561 ts_manager.cc:194] Re-registered known tserver with Master: 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143)
I20260925 05:06:00.466470 20561 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:36213
W20260925 05:06:00.539749 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:00.539944 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:00.539976 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:00.540977 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:00.541059 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:00.541100 21533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:00.542502 21533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:39747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=33737
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:00.542738 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:00.542969 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.547401 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.547411 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.547394 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.547580 21533 server_base.cc:1034] running on GCE node
I20260925 05:06:00.547770 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.547981 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.549170 21533 hybrid_clock.cc:648] HybridClock initialized: now 1790312760549156 us; error 39 us; skew 500 ppm
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.551335 21533 init.cc:377] Logged in from keytab as oryx/127.14.33.2@KRBTEST.COM (short username oryx)
I20260925 05:06:00.551702 21533 webserver.cc:460] Webserver started at http://127.14.33.2:33737/ using document root <none> and password file <none>
I20260925 05:06:00.551901 21533 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.551970 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.553175 21533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:00.553752 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.553908 21533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:00.553992 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "b9acf3f3d9af4931ba9623540c32ddfa"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "f3ce0a81bffd0bc2d8b0f693d2910b75"
server_key_iv: "b4084189b2663d5907799609d32f56f7"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.554355 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.577816 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.579665 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:00.579821 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.580037 21533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:00.580319 21533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:00.580365 21533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.580411 21533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:00.580435 21533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.585807 21533 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:39747
I20260925 05:06:00.585870 21661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:39747 every 8 connection(s)
I20260925 05:06:00.586164 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:00.587868 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 21533
I20260925 05:06:00.588017 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20891
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.60@KRBTEST.COM
I20260925 05:06:00.594496 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:43701
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:39543
--webserver_port=40235
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:00.597329 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.587150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:36495 (local address 127.14.33.62:39543)
0925 05:06:00.587232 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.587234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.587246 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:06:00.587569 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.587570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.587583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.587631 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.589021 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.589775 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.590876 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.591176 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591216 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.593751 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.593753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.593753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.593760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.595348 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.596287 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.596288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.596288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.596348 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.596634 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.596636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.596636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.596765 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.596831 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.597144 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.597231 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:06:00.597499 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.586745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:36495)
0925 05:06:00.586880 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.586915 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.587463 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.587706 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.587710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.588863 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.588930 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.588932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.589866 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.589867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.590789 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.590791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.591301 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.591388 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.591877 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.591886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.593651 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.595428 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.595435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.595437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.596205 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.596427 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.596428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.596428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.596559 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.596836 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.596841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.596982 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.597082 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.597272 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260925 05:06:00.598071 21663 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39543
I20260925 05:06:00.598153 21663 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.598332 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.587250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:49759 (local address 127.14.33.60:37515)
0925 05:06:00.587306 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.587307 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.587315 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:00.587574 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.587575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.587586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.587618 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.588626 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.589342 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591315 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.591590 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591628 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.595587 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.595589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.595589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.595596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.597038 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.597415 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.597417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.597417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.597472 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.597647 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.597648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.597648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.597773 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.597832 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.598111 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.598195 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:06:00.598346 21663 heartbeater.cc:511] Master 127.14.33.62:39543 requested a full tablet report, sending...
I20260925 05:06:00.598516 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.586735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:55241 (local address 127.14.33.61:33121)
0925 05:06:00.586778 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.586780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.586987 (+ 207us) server_negotiation.cc:407] Connection header received
0925 05:06:00.587574 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.587575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.587586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.587618 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.588627 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.589344 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.590876 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.591132 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591167 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.594837 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.594839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.594840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.594846 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.596224 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.597777 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.597778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.597778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.597832 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.598015 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.598016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.598016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.598131 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.598173 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.598371 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.598436 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:06:00.598841 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.586745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.14.33.2:55241)
0925 05:06:00.586906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.586916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.587483 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.587712 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.587713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.588487 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.588536 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.588537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.589471 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.589472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.590790 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.590793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.591248 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.591386 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.592149 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.592154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.594741 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.596332 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.596332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.596333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.597706 (+ 1373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.597899 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.597900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.597900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.597956 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.598234 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.598234 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.598271 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.598613 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.598732 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":44,"spinlock_wait_cycles":384,"thread_start_us":103,"threads_started":1}
I20260925 05:06:00.598904 20561 ts_manager.cc:194] Re-registered known tserver with Master: b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747)
I20260925 05:06:00.599234 21664 heartbeater.cc:347] Connected to a master server at 127.14.33.61:33121
I20260925 05:06:00.599270 21664 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.599319 21664 heartbeater.cc:511] Master 127.14.33.61:33121 requested a full tablet report, sending...
I20260925 05:06:00.599381 20561 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:36495
I20260925 05:06:00.599709 21158 ts_manager.cc:194] Registered new tserver with Master: b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747)
I20260925 05:06:00.600085 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.587002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.14.33.2:49759)
0925 05:06:00.587119 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.587129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.587479 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.588063 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.588064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.588222 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.588523 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.588529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.589471 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.589472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.591218 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.591221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.591701 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.591702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.591773 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.592417 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.592439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.595520 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.597125 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.597125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.597126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.597342 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.597550 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.597551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.597551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.597598 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.597856 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.597857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.597894 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.597946 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.598036 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":726,"thread_start_us":62,"threads_started":1}
I20260925 05:06:00.600461 21662 heartbeater.cc:347] Connected to a master server at 127.14.33.60:37515
I20260925 05:06:00.600518 21662 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.600612 21662 heartbeater.cc:511] Master 127.14.33.60:37515 requested a full tablet report, sending...
I20260925 05:06:00.600939 21345 ts_manager.cc:194] Registered new tserver with Master: b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747)
W20260925 05:06:00.671887 21669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:00.672045 21669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:00.672063 21669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:00.673014 21669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:00.673044 21669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:00.673084 21669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:00.674387 21669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:43701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=40235
--tserver_master_addrs=127.14.33.62:39543,127.14.33.61:33121,127.14.33.60:37515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:00.674552 21669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:00.674693 21669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:00.679032 21675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.679060 21677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:00.679077 21674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:00.679306 21669 server_base.cc:1034] running on GCE node
I20260925 05:06:00.679469 21669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:00.679692 21669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:00.680822 21669 hybrid_clock.cc:648] HybridClock initialized: now 1790312760680814 us; error 26 us; skew 500 ppm
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:00.683143 21669 init.cc:377] Logged in from keytab as oryx/127.14.33.3@KRBTEST.COM (short username oryx)
I20260925 05:06:00.683504 21669 webserver.cc:460] Webserver started at http://127.14.33.3:40235/ using document root <none> and password file <none>
I20260925 05:06:00.683699 21669 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:00.683749 21669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:00.684885 21669 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:00.685389 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.685561 21669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.685637 21669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "294bef61bf664c5689c561d6e8bec574"
format_stamp: "Formatted at 2026-09-25 05:05:59 on dist-test-slave-4dmq"
server_key: "c81e1e6d529ba529c1938575804d927d"
server_key_iv: "ae791607a525e2aec7610b635d37cba5"
server_key_version: "encryptionkey@0"
I20260925 05:06:00.685884 21669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:00.711947 21669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:00.713649 21669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:00.713744 21669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:00.713900 21669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:00.714167 21669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:00.714197 21669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.714217 21669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:00.714270 21669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:00.719923 21669 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:43701
I20260925 05:06:00.720018 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:43701 every 8 connection(s)
I20260925 05:06:00.720327 21669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.61@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.60@KRBTEST.COM
Sep 25 05:06:00 dist-test-slave-4dmq krb5kdc[20516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:00.730204 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 21669
I20260925 05:06:00.731056 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.720993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:33981 (local address 127.14.33.60:37515)
0925 05:06:00.721042 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.721044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.721343 (+ 299us) server_negotiation.cc:407] Connection header received
0925 05:06:00.722432 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.722434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.722448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.722482 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.723093 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.723812 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.724920 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.725210 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.725286 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.727611 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.727613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.727613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.727621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.729698 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.730171 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.730172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.730173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.730223 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.730402 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.730404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.730404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.730561 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.730624 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.730900 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.730976 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:06:00.731096 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.721382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:32947 (local address 127.14.33.61:33121)
0925 05:06:00.721433 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.721435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.721465 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:06:00.722402 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.722402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.722413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.722454 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.723064 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.723503 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.724490 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.724659 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.724743 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.728034 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.728035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.728036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.728043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.729581 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.730051 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.730052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.730052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.730125 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.730397 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.730398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.730399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.730515 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.730544 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.730935 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.731003 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:06:00.731242 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.721007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.14.33.3:33981)
0925 05:06:00.721205 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.721252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.722304 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.722551 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.722551 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.722888 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.722982 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.722992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.723925 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.723925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.724844 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.724845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.725332 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.725332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.725395 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.725889 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.725893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.727506 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.729779 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.729780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.729781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.730090 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.730291 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.730291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.730292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.730336 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.730631 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.730632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.730760 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.730873 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.731005 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":74,"thread_start_us":85,"threads_started":1}
I20260925 05:06:00.731398 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.721300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.14.33.3:32947)
0925 05:06:00.721413 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.721422 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.722319 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.722516 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.722519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.722813 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.722981 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.722985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.723596 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.723597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.724395 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.724397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.724756 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.724756 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.724872 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.725580 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.725596 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.727918 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.729671 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.729677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.729679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.729979 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.730202 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.730202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.730203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.730324 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.730575 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.730580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.730759 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.731174 (+ 415us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.731263 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260925 05:06:00.731616 21798 heartbeater.cc:347] Connected to a master server at 127.14.33.60:37515
I20260925 05:06:00.731722 21798 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.731890 21798 heartbeater.cc:511] Master 127.14.33.60:37515 requested a full tablet report, sending...
I20260925 05:06:00.732043 21800 heartbeater.cc:347] Connected to a master server at 127.14.33.61:33121
I20260925 05:06:00.732100 21800 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.732167 21800 heartbeater.cc:511] Master 127.14.33.61:33121 requested a full tablet report, sending...
I20260925 05:06:00.732352 21345 ts_manager.cc:194] Registered new tserver with Master: 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701)
I20260925 05:06:00.732471 21158 ts_manager.cc:194] Registered new tserver with Master: 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701)
I20260925 05:06:00.732861 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.720993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:39177 (local address 127.14.33.62:39543)
0925 05:06:00.721042 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.721044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.721343 (+ 299us) server_negotiation.cc:407] Connection header received
0925 05:06:00.722433 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.722434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.722448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.722482 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.723094 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.723803 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.724947 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.725252 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.725330 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.728276 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.728277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.728278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.728285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.729856 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.731307 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.731309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.731309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.731367 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.731542 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.731543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.731543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.731647 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.731686 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.732646 (+ 960us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.732746 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:06:00.733318 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.721007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:39177)
0925 05:06:00.721205 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.721252 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.722304 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.722550 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.722551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.722885 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:00.722982 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.722992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.723925 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.723925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.724837 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.724840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.725330 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.725331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.725395 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.725952 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:00.725957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:00.727917 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:00.729948 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.729949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.729950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.731214 (+ 1264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.731424 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:00.731425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:00.731425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:00.731472 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:00.731741 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:00.731741 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:00.731790 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.733014 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.733154 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":51,"spinlock_wait_cycles":7040,"thread_start_us":119,"threads_started":1}
I20260925 05:06:00.733623 21799 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39543
I20260925 05:06:00.733678 21799 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:00.733752 21799 heartbeater.cc:511] Master 127.14.33.62:39543 requested a full tablet report, sending...
I20260925 05:06:00.733973 20561 ts_manager.cc:194] Re-registered known tserver with Master: 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701)
I20260925 05:06:00.734334 20561 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:39177
I20260925 05:06:00.740352 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.732652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.14.33.61:33121)
0925 05:06:00.732690 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.732691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.733287 (+ 596us) server_negotiation.cc:407] Connection header received
0925 05:06:00.733427 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.733427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.733438 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.733467 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.734161 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.734847 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.735825 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.736039 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.736112 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.737516 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.737517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.737517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.737523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.739048 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.739523 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.739524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.739525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.739578 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.739804 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.739804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.739805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.739925 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.739992 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.740201 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.740277 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:06:00.740437 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.733048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.14.33.60:37515)
0925 05:06:00.733106 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.733108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.733703 (+ 595us) server_negotiation.cc:407] Connection header received
0925 05:06:00.733797 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.733798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.733808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.733860 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.734423 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.735034 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.735968 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.736233 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.736314 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.737857 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.737858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.737858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.737865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.739201 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.739654 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.739655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.739656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.739709 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.739914 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.739915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.739916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.740031 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.740121 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.740282 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.740346 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:06:00.742317 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.732215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.14.33.62:39543)
0925 05:06:00.732950 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.732952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.732973 (+ 21us) server_negotiation.cc:407] Connection header received
0925 05:06:00.733217 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.733218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.733228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.733281 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.736250 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.737099 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.737890 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.738166 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.738205 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.739669 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.739670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.739670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.739677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.741378 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.741653 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.741654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.741654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.741699 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.741880 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.741881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.741881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.741976 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.742069 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.742147 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.742224 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":77,"threads_started":1}
I20260925 05:06:00.743602 20561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:00.745278 20561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:00.752125 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.748415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39143 (local address 127.0.0.1:58404)
0925 05:06:00.748468 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.748477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.748519 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.749313 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.749313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.749316 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.749377 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.749379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.750208 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.750209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.751064 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.751066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.751906 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.751907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.751965 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.751969 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.752030 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.752066 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":8}
I20260925 05:06:00.752333 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.748887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.14.33.1:39143)
0925 05:06:00.749068 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.749073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.749082 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:00.749174 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.749176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.749192 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.749434 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.749478 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.750105 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.751151 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.751827 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.751894 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.751960 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.752043 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.752081 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260925 05:06:00.752434 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.748508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:43701 (local address 127.0.0.1:36862)
0925 05:06:00.748588 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.748595 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.748621 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.749313 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.749314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.749316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.749368 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.749370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.750578 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.750579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.751551 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.751553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.752253 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.752254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.752308 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.752311 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.752337 (+ 26us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.752372 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":25}
I20260925 05:06:00.752712 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.748711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.14.33.3:43701)
0925 05:06:00.749093 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.749097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.749107 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:06:00.749167 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.749169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.749187 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.749436 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.749475 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.750490 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.751619 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.752166 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.752531 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.752601 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.752623 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.752654 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":79,"threads_started":1}
I20260925 05:06:00.752935 21460 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9 (DEFAULT_TABLE table=test-table [id=7f29f526ec8542d4857efcd7499dc321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:00.753284 21460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:00.753818 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9 (DEFAULT_TABLE table=test-table [id=7f29f526ec8542d4857efcd7499dc321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:00.754118 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:00.754330 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.748616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39747 (local address 127.0.0.1:41042)
0925 05:06:00.748742 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.748753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.748801 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.749475 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.749476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.749477 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.749512 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.749514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.750654 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.750655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.752648 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.752651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.754093 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.754094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.754159 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.754163 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.754228 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.754269 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260925 05:06:00.754542 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.748724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.14.33.2:39747)
0925 05:06:00.749242 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.749246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.749256 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:06:00.749310 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.749312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.749326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.749406 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.749588 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.750581 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.752728 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.753928 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.753997 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.754056 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.754430 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.754468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":77,"threads_started":1}
I20260925 05:06:00.755259 21596 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9 (DEFAULT_TABLE table=test-table [id=7f29f526ec8542d4857efcd7499dc321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:00.755576 21596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:00.756022 21821 tablet_bootstrap.cc:492] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Bootstrap starting.
I20260925 05:06:00.756707 21821 tablet_bootstrap.cc:654] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:00.756974 21821 log.cc:824] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:00.757696 21821 tablet_bootstrap.cc:492] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: No bootstrap required, opened a new log
I20260925 05:06:00.757802 21821 ts_tablet_manager.cc:1397] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:00.758112 21823 tablet_bootstrap.cc:492] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Bootstrap starting.
I20260925 05:06:00.758633 21823 tablet_bootstrap.cc:654] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:00.758962 21823 log.cc:824] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:00.759279 21824 tablet_bootstrap.cc:492] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Bootstrap starting.
I20260925 05:06:00.759490 21821 raft_consensus.cc:348] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.759631 21823 tablet_bootstrap.cc:492] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: No bootstrap required, opened a new log
I20260925 05:06:00.759666 21821 raft_consensus.cc:374] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:00.759701 21821 raft_consensus.cc:729] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1d9d241f924f2384681a943df80a18, State: Initialized, Role: FOLLOWER
I20260925 05:06:00.759716 21823 ts_tablet_manager.cc:1397] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:00.759825 21821 consensus_queue.cc:260] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.759915 21824 tablet_bootstrap.cc:654] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:00.760092 21821 ts_tablet_manager.cc:1428] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:06:00.760213 21527 heartbeater.cc:503] Master 127.14.33.62:39543 was elected leader, sending a full tablet report...
I20260925 05:06:00.760972 21824 log.cc:824] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:00.761888 21824 tablet_bootstrap.cc:492] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: No bootstrap required, opened a new log
I20260925 05:06:00.761936 21824 ts_tablet_manager.cc:1397] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 05:06:00.762072 21823 raft_consensus.cc:348] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.762202 21823 raft_consensus.cc:374] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:00.762240 21823 raft_consensus.cc:729] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294bef61bf664c5689c561d6e8bec574, State: Initialized, Role: FOLLOWER
I20260925 05:06:00.762358 21823 consensus_queue.cc:260] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.762703 21823 ts_tablet_manager.cc:1428] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:06:00.762878 21799 heartbeater.cc:503] Master 127.14.33.62:39543 was elected leader, sending a full tablet report...
I20260925 05:06:00.763974 21824 raft_consensus.cc:348] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.764084 21824 raft_consensus.cc:374] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:00.764104 21824 raft_consensus.cc:729] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9acf3f3d9af4931ba9623540c32ddfa, State: Initialized, Role: FOLLOWER
I20260925 05:06:00.764202 21824 consensus_queue.cc:260] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.764391 21824 ts_tablet_manager.cc:1428] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:06:00.764456 21663 heartbeater.cc:503] Master 127.14.33.62:39543 was elected leader, sending a full tablet report...
I20260925 05:06:00.773100 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.770776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.14.33.60:37515)
0925 05:06:00.770812 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.770814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.770821 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:00.770840 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.770841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.770858 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.770913 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.771001 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.771438 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.772382 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.772865 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.772921 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.772961 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.773033 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.773056 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:06:00.773105 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.770689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40196)
0925 05:06:00.770743 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.770754 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.770793 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.770921 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.770922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.770923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.770957 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.770959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.771515 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.771516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.772237 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.772239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.772930 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.772931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.772977 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.772980 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.773027 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.773060 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:06:00.773474 21355 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 05:06:00.774178 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda2cc968b064952b2670248d6ffdfca. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.774271 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: This master's current role is: LEARNER
I20260925 05:06:00.819989 21829 raft_consensus.cc:482] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:00.820096 21829 raft_consensus.cc:504] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.820398 21829 leader_election.cc:288] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143), 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701)
I20260925 05:06:00.823113 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.820728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:43701 (local address 127.14.33.2:42711)
0925 05:06:00.820770 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.820778 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.820813 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.820987 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.820988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.820989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.821064 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.821066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.821720 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.821721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.822385 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.822387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.822894 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.822895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.822962 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.822967 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.823016 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.823056 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260925 05:06:00.823168 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.820801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:42711 (local address 127.14.33.3:43701)
0925 05:06:00.820844 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.820846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.820855 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:00.820900 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.820901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.820912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.820980 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.821182 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.821645 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.822475 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.822820 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.822880 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.822888 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.823039 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.823074 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:06:00.823285 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.820592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39143 (local address 127.14.33.2:50191)
0925 05:06:00.820683 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:00.820697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:00.820753 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:00.820906 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:00.820907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:00.820909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:00.820991 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.820993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.821615 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.821616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.822385 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:00.822388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.823041 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.823043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:00.823109 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.823113 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:06:00.823174 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:06:00.823210 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":17}
I20260925 05:06:00.823434 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2859fdfb0b47d783c6cd09b2f9a5d9" candidate_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294bef61bf664c5689c561d6e8bec574" is_pre_election: true
I20260925 05:06:00.823624 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.820691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:50191 (local address 127.14.33.1:39143)
0925 05:06:00.820741 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.820743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.820752 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:00.820820 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.820821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.820833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.820887 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:00.821064 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.821534 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.822499 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.822944 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.823011 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.823021 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.823526 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.823568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:06:00.823632 21752 raft_consensus.cc:2454] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9acf3f3d9af4931ba9623540c32ddfa in term 0.
I20260925 05:06:00.823858 21549 leader_election.cc:302] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294bef61bf664c5689c561d6e8bec574, b9acf3f3d9af4931ba9623540c32ddfa; no voters:
I20260925 05:06:00.823997 21829 raft_consensus.cc:2781] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:00.824052 21829 raft_consensus.cc:482] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:00.824074 21829 raft_consensus.cc:3037] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:00.824074 21480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2859fdfb0b47d783c6cd09b2f9a5d9" candidate_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c1d9d241f924f2384681a943df80a18" is_pre_election: true
I20260925 05:06:00.824227 21480 raft_consensus.cc:2454] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9acf3f3d9af4931ba9623540c32ddfa in term 0.
I20260925 05:06:00.824869 21829 raft_consensus.cc:504] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.824986 21829 leader_election.cc:288] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [CANDIDATE]: Term 1 election: Requested vote from peers 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143), 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701)
I20260925 05:06:00.825167 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2859fdfb0b47d783c6cd09b2f9a5d9" candidate_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "294bef61bf664c5689c561d6e8bec574"
I20260925 05:06:00.825165 21480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2859fdfb0b47d783c6cd09b2f9a5d9" candidate_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c1d9d241f924f2384681a943df80a18"
I20260925 05:06:00.825244 21752 raft_consensus.cc:3037] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:00.825264 21480 raft_consensus.cc:3037] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:00.825870 21752 raft_consensus.cc:2454] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9acf3f3d9af4931ba9623540c32ddfa in term 1.
I20260925 05:06:00.826011 21480 raft_consensus.cc:2454] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9acf3f3d9af4931ba9623540c32ddfa in term 1.
I20260925 05:06:00.826020 21549 leader_election.cc:302] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294bef61bf664c5689c561d6e8bec574, b9acf3f3d9af4931ba9623540c32ddfa; no voters:
I20260925 05:06:00.826192 21829 raft_consensus.cc:2781] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:00.826359 21829 raft_consensus.cc:686] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 LEADER]: Becoming Leader. State: Replica: b9acf3f3d9af4931ba9623540c32ddfa, State: Running, Role: LEADER
I20260925 05:06:00.826467 21829 consensus_queue.cc:237] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } }
I20260925 05:06:00.827205 20561 catalog_manager.cc:5495] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa reported cstate change: term changed from 0 to 1, leader changed from <none> to b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2). New cstate: current_term: 1 leader_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:00.827771 21119 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: NON_VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: true } } }
I20260925 05:06:00.827849 21119 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.828646 21308 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: attempting to promote NON_VOTER 19fe7dc2d89e41e0ba921f3524f590a7 to VOTER
I20260925 05:06:00.828769 21308 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:00.829137 21355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 LEARNER]: Refusing update from remote peer eda2cc968b064952b2670248d6ffdfca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 05:06:00.829270 21168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Refusing update from remote peer eda2cc968b064952b2670248d6ffdfca: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 05:06:00.829304 21119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 05:06:00.829497 21119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 05:06:00.829973 21835 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.830117 21355 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.831099 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.831216 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.831208 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.831274 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.831307 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:00.831318 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.831295 21119 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.831396 21119 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:00.831548 21168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.832301 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eda2cc968b064952b2670248d6ffdfca" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:00.832407 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:06:00.837288 21665 tablet.cc:2369] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:00.846622 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.844545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.14.33.2:39747)
0925 05:06:00.844580 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.844581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.844590 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:00.844656 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.844657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.844678 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.844740 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:00.844875 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.845295 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.845979 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.846148 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.846215 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.846357 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:00.846440 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:00.846513 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.846547 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.846565 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:06:00.848052 21480 raft_consensus.cc:1261] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 1 FOLLOWER]: Refusing update from remote peer b9acf3f3d9af4931ba9623540c32ddfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:00.848052 21752 raft_consensus.cc:1261] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Refusing update from remote peer b9acf3f3d9af4931ba9623540c32ddfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:00.848377 21829 consensus_queue.cc:1035] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:00.848539 21833 consensus_queue.cc:1035] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:00.852622 21840 mvcc.cc:205] Tried to move back new op lower bound from 7333121068431196160 to 7333121068345794560. Current Snapshot: MvccSnapshot[applied={T|T < 7333121068431196160}]
I20260925 05:06:00.854807 20561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:00.854887 20561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:00.856187 20561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eda2cc968b064952b2670248d6ffdfca: Sending DeleteTablet for 3 replicas of tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9
I20260925 05:06:00.856539 21596 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:00 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:41042
I20260925 05:06:00.856544 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:00 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:36862
I20260925 05:06:00.856739 21460 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2859fdfb0b47d783c6cd09b2f9a5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:00 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:58404
I20260925 05:06:00.856776 21849 tablet_replica.cc:330] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: stopping tablet replica
I20260925 05:06:00.856869 21849 raft_consensus.cc:2229] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:00.857004 21849 raft_consensus.cc:2258] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:00.857223 21850 tablet_replica.cc:330] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: stopping tablet replica
I20260925 05:06:00.857309 21850 raft_consensus.cc:2229] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:00.857404 21850 raft_consensus.cc:2258] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:00.857520 21849 ts_tablet_manager.cc:1905] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:00.857820 21850 ts_tablet_manager.cc:1905] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:00.857914 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 20529
I20260925 05:06:00.859776 21850 ts_tablet_manager.cc:1918] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:00.859865 21850 log.cc:1196] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/2c2859fdfb0b47d783c6cd09b2f9a5d9
I20260925 05:06:00.860126 21850 ts_tablet_manager.cc:1939] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 294bef61bf664c5689c561d6e8bec574: Deleting consensus metadata
I20260925 05:06:00.862208 21849 ts_tablet_manager.cc:1918] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:00.862301 21849 log.cc:1196] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/2c2859fdfb0b47d783c6cd09b2f9a5d9
I20260925 05:06:00.862558 21849 ts_tablet_manager.cc:1939] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P b9acf3f3d9af4931ba9623540c32ddfa: Deleting consensus metadata
W20260925 05:06:00.864969 21686 connection.cc:535] server connection from 127.0.0.1:36862 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 05:06:00.865061 21551 connection.cc:535] server connection from 127.0.0.1:41042 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:06:00.865252 21851 tablet_replica.cc:330] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: stopping tablet replica
I20260925 05:06:00.865338 21851 raft_consensus.cc:2229] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:00.865432 21851 raft_consensus.cc:2258] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:00.865953 21851 ts_tablet_manager.cc:1905] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:00.867750 21851 ts_tablet_manager.cc:1918] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:00.867832 21851 log.cc:1196] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/2c2859fdfb0b47d783c6cd09b2f9a5d9
I20260925 05:06:00.868166 21851 ts_tablet_manager.cc:1939] T 2c2859fdfb0b47d783c6cd09b2f9a5d9 P 2c1d9d241f924f2384681a943df80a18: Deleting consensus metadata
W20260925 05:06:00.868535 21414 connection.cc:441] server connection from 127.0.0.1:58404 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58404 (request call id 1) could send its response
I20260925 05:06:00.875195 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.867389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.14.33.60:37515)
0925 05:06:00.867435 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.867437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.868597 (+ 1160us) server_negotiation.cc:407] Connection header received
0925 05:06:00.868715 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.868716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.868727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.868787 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.869517 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.869971 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.870893 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.871172 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.871265 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.872546 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.872547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.872547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.872554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.873873 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.874269 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.874271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.874271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.874333 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.874609 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.874610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.874610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.874747 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.874842 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.875028 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.875101 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:06:00.875901 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:00.867159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.14.33.61:33121)
0925 05:06:00.867202 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:00.867204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:00.867369 (+ 165us) server_negotiation.cc:407] Connection header received
0925 05:06:00.867480 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:00.867481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:00.867493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:00.867527 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:00.868415 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.869187 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.870180 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:00.870438 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:00.870511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:00.872247 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:00.872248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:00.872249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:00.872257 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:00.874442 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.874811 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.874812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.874813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.874871 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:00.875092 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:00.875093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:00.875094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:00.875221 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:00.875339 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:00.875725 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 05:06:00.875806 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:06:01.852669 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:01.852365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:50527)
0925 05:06:01.852519 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:01.852606 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20260925 05:06:01.853161 21527 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:39543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:01.853312 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:01.852925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:58067)
0925 05:06:01.853193 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:01.853265 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":45,"threads_started":1}
I20260925 05:06:01.853334 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:01.852507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:43715)
0925 05:06:01.853234 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:01.853295 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":57,"threads_started":1}
W20260925 05:06:01.853499 21799 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:39543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
W20260925 05:06:01.853555 21663 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:39543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.357020 21862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eda2cc968b064952b2670248d6ffdfca)
I20260925 05:06:02.357013 21861 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eda2cc968b064952b2670248d6ffdfca)
I20260925 05:06:02.357188 21862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:02.357189 21861 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:02.357465 21862 leader_election.cc:288] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543), f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61:33121)
I20260925 05:06:02.357488 21861 leader_election.cc:288] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543), 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60:37515)
I20260925 05:06:02.357818 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.357547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41698)
0925 05:06:02.357729 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.357772 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":51,"threads_started":1}
W20260925 05:06:02.358021 21329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.358130 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.357665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41706)
0925 05:06:02.358038 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.358092 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":69,"threads_started":1}
W20260925 05:06:02.358223 21142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.358543 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.358454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41712)
0925 05:06:02.358488 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.358521 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 05:06:02.358669 21329 leader_election.cc:334] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.358700 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.358610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41726)
0925 05:06:02.358643 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.358663 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 05:06:02.358755 21142 leader_election.cc:334] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.360527 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.357605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:33121 (local address 127.0.0.1:49946)
0925 05:06:02.357900 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.357909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:02.357961 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:02.358113 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:02.358114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:02.358116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:02.358165 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:02.358167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.359083 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.359083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:02.359769 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:02.359770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.360306 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.360307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:02.360369 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:02.360371 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:06:02.360412 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 05:06:02.360442 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":46,"threads_started":1}
I20260925 05:06:02.360615 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.357838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.14.33.60:37515)
0925 05:06:02.357929 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:02.357931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:02.357939 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:02.357963 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:02.357964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:02.357973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:02.358036 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:02.358226 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.358972 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.359880 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.360242 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.360326 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:02.360336 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:02.360490 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:06:02.360522 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
I20260925 05:06:02.360546 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.357766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.14.33.61:33121)
0925 05:06:02.357843 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:02.357845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:02.357947 (+ 102us) server_negotiation.cc:407] Connection header received
0925 05:06:02.358006 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:02.358007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:02.358016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:02.358068 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:02.358270 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.358985 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.359850 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.360233 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.360285 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:02.360315 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:02.360431 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 05:06:02.360476 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"thread_start_us":29,"threads_started":1}
I20260925 05:06:02.360818 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.357651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:37515 (local address 127.0.0.1:40216)
0925 05:06:02.357733 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.357745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:02.357836 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:02.358089 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:02.358090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:02.358093 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:02.358149 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:02.358151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.359048 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.359049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:02.359775 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:02.359777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:02.360338 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:02.360338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:02.360416 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:02.360420 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:06:02.360475 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:06:02.360509 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":38,"threads_started":1}
I20260925 05:06:02.360841 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f25eb31273c0475fa6f42d55fa592f73" is_pre_election: true
I20260925 05:06:02.360970 21168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fe7dc2d89e41e0ba921f3524f590a7 in term 1.
I20260925 05:06:02.361025 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f25eb31273c0475fa6f42d55fa592f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" is_pre_election: true
I20260925 05:06:02.361109 21332 leader_election.cc:302] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19fe7dc2d89e41e0ba921f3524f590a7, f25eb31273c0475fa6f42d55fa592f73; no voters: eda2cc968b064952b2670248d6ffdfca
I20260925 05:06:02.361187 21355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f25eb31273c0475fa6f42d55fa592f73 in term 1.
I20260925 05:06:02.361249 21862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 05:06:02.361294 21862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eda2cc968b064952b2670248d6ffdfca)
I20260925 05:06:02.361313 21862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:06:02.361330 21145 leader_election.cc:302] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19fe7dc2d89e41e0ba921f3524f590a7, f25eb31273c0475fa6f42d55fa592f73; no voters: eda2cc968b064952b2670248d6ffdfca
I20260925 05:06:02.361438 21861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 05:06:02.361474 21861 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eda2cc968b064952b2670248d6ffdfca)
I20260925 05:06:02.361493 21861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:06:02.362219 21862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:02.362354 21862 leader_election.cc:288] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 2 election: Requested vote from peers eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543), f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61:33121)
I20260925 05:06:02.362319 21861 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:02.362426 21861 leader_election.cc:288] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [CANDIDATE]: Term 2 election: Requested vote from peers eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543), 19fe7dc2d89e41e0ba921f3524f590a7 (127.14.33.60:37515)
I20260925 05:06:02.362593 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f25eb31273c0475fa6f42d55fa592f73"
I20260925 05:06:02.362573 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f25eb31273c0475fa6f42d55fa592f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19fe7dc2d89e41e0ba921f3524f590a7"
I20260925 05:06:02.362592 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.362435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41740)
0925 05:06:02.362536 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.362561 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 05:06:02.362670 21168 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 19fe7dc2d89e41e0ba921f3524f590a7 in current term 2: Already voted for candidate f25eb31273c0475fa6f42d55fa592f73 in this term.
I20260925 05:06:02.362689 21355 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f25eb31273c0475fa6f42d55fa592f73 in current term 2: Already voted for candidate 19fe7dc2d89e41e0ba921f3524f590a7 in this term.
I20260925 05:06:02.362859 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.362495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41742)
0925 05:06:02.362628 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.362669 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 05:06:02.362927 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.362827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41746)
0925 05:06:02.362872 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.362900 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:06:02.362987 21329 leader_election.cc:334] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.363039 21332 leader_election.cc:302] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fe7dc2d89e41e0ba921f3524f590a7; no voters: eda2cc968b064952b2670248d6ffdfca, f25eb31273c0475fa6f42d55fa592f73
I20260925 05:06:02.363127 21862 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 05:06:02.363226 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.363146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:41756)
0925 05:06:02.363177 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.363205 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 05:06:02.363333 21142 leader_election.cc:334] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:02.363376 21142 leader_election.cc:302] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f25eb31273c0475fa6f42d55fa592f73; no voters: 19fe7dc2d89e41e0ba921f3524f590a7, eda2cc968b064952b2670248d6ffdfca
I20260925 05:06:02.363530 21861 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 05:06:02.853987 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.853764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:52239)
0925 05:06:02.853891 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.853927 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260925 05:06:02.854030 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.853643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:52749)
0925 05:06:02.853941 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.853982 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":75,"threads_started":1}
I20260925 05:06:02.854084 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:02.853851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:46769)
0925 05:06:02.854014 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:02.854041 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":54,"threads_started":1}
I20260925 05:06:03.854595 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:03.854371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:38667)
0925 05:06:03.854498 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:03.854539 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260925 05:06:03.854760 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:03.854431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:57721)
0925 05:06:03.854666 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:03.854714 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":78,"threads_started":1}
I20260925 05:06:03.854790 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:03.854626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:34223)
0925 05:06:03.854723 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:03.854752 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":43,"threads_started":1}
I20260925 05:06:04.191817 21877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:04.191898 21877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:04.192076 21877 leader_election.cc:288] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543), f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61:33121)
I20260925 05:06:04.192253 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f25eb31273c0475fa6f42d55fa592f73" is_pre_election: true
I20260925 05:06:04.192333 21168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fe7dc2d89e41e0ba921f3524f590a7 in term 2.
I20260925 05:06:04.192342 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.192176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:42032)
0925 05:06:04.192266 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.192295 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":58,"threads_started":1}
I20260925 05:06:04.192478 21332 leader_election.cc:302] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fe7dc2d89e41e0ba921f3524f590a7, f25eb31273c0475fa6f42d55fa592f73; no voters:
I20260925 05:06:04.192556 21877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 05:06:04.192601 21877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:04.192665 21877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:06:04.192699 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.192621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:42042)
0925 05:06:04.192652 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.192675 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 05:06:04.192781 21329 leader_election.cc:334] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:04.193511 21877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:04.193671 21877 leader_election.cc:288] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 3 election: Requested vote from peers eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543), f25eb31273c0475fa6f42d55fa592f73 (127.14.33.61:33121)
I20260925 05:06:04.193861 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.193762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:42050)
0925 05:06:04.193801 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.193835 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 05:06:04.193826 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f25eb31273c0475fa6f42d55fa592f73"
I20260925 05:06:04.193904 21168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:06:04.194170 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.194059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:42052)
0925 05:06:04.194117 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.194148 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 05:06:04.194252 21329 leader_election.cc:334] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
I20260925 05:06:04.194690 21168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fe7dc2d89e41e0ba921f3524f590a7 in term 3.
I20260925 05:06:04.194868 21332 leader_election.cc:302] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19fe7dc2d89e41e0ba921f3524f590a7, f25eb31273c0475fa6f42d55fa592f73; no voters: eda2cc968b064952b2670248d6ffdfca
I20260925 05:06:04.194980 21877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:06:04.195101 21877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [term 3 LEADER]: Becoming Leader. State: Replica: 19fe7dc2d89e41e0ba921f3524f590a7, State: Running, Role: LEADER
I20260925 05:06:04.195190 21877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } }
I20260925 05:06:04.195533 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19fe7dc2d89e41e0ba921f3524f590a7. Latest consensus state: current_term: 3 leader_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:04.195631 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:04.195792 21882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:04.196159 21882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:04.196241 21882 catalog_manager.cc:1259] Loaded cluster ID: e10689a784b4439080aa2e998ce35e19
I20260925 05:06:04.196277 21882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:04.196449 21882 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:04.196537 21882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Loaded TSK: 0
I20260925 05:06:04.196671 21882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:04.273288 21345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:04.274837 21345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:04.275854 21168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [term 3 FOLLOWER]: Refusing update from remote peer 19fe7dc2d89e41e0ba921f3524f590a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 05:06:04.275884 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.275767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:42062)
0925 05:06:04.275819 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.275850 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:06:04.276055 21877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 05:06:04.276211 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.276087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.0.0.1:42076)
0925 05:06:04.276133 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.276163 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:06:04.276278 21329 consensus_peers.cc:487] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 -> Peer eda2cc968b064952b2670248d6ffdfca (127.14.33.62:39543): Couldn't send request to peer eda2cc968b064952b2670248d6ffdfca. Status: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 05:06:04.276609 21884 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19fe7dc2d89e41e0ba921f3524f590a7. Latest consensus state: current_term: 3 leader_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:04.276708 21884 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:04.277137 21885 mvcc.cc:205] Tried to move back new op lower bound from 7333121082471776256 to 7333121082144075776. Current Snapshot: MvccSnapshot[applied={T|T < 7333121082471776256}]
I20260925 05:06:04.277118 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:04.277212 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:04.277369 21884 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eda2cc968b064952b2670248d6ffdfca" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39543 } } peers { permanent_uuid: "f25eb31273c0475fa6f42d55fa592f73" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 33121 } attrs { promote: false } } peers { permanent_uuid: "19fe7dc2d89e41e0ba921f3524f590a7" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 37515 } attrs { promote: false } } }
I20260925 05:06:04.277448 21884 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25eb31273c0475fa6f42d55fa592f73 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:06:04.281858 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.279121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:43701 (local address 127.0.0.1:36876)
0925 05:06:04.279216 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.279224 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:04.279265 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:04.279604 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:04.279605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:04.279607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:04.279643 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.279644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.280382 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.280383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.281011 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.281013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.281625 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.281626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.281686 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.281689 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:04.281741 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:06:04.281778 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260925 05:06:04.281859 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.279391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.14.33.3:43701)
0925 05:06:04.279477 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.279479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.279485 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:06:04.279506 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.279507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.279515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.279693 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:04.279725 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.280304 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.281093 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.281545 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.281627 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.281638 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.281758 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.281791 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":41,"threads_started":1}
I20260925 05:06:04.282017 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.278925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39747 (local address 127.0.0.1:41064)
0925 05:06:04.278972 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.278982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:04.279063 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:04.279230 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:04.279231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:04.279233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:04.279296 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.279298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.280156 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.280157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.280809 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.280810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.281823 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.281824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.281878 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.281882 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:06:04.281910 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 05:06:04.281954 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:06:04.282230 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.278989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.14.33.2:39747)
0925 05:06:04.279087 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.279089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.279098 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:04.279123 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.279124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.279135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.279361 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:04.279401 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.280072 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.280903 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.281737 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.282081 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.282095 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.282114 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.282152 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260925 05:06:04.282444 21596 tablet_service.cc:1467] Processing CreateTablet for tablet 6859cb70bb0140bb8f565efac6dfac84 (DEFAULT_TABLE table=test-table [id=6f67e299b7c247aebe1f2bf0e5d3bd1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:04.282568 21596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6859cb70bb0140bb8f565efac6dfac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:04.282838 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.278971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39143 (local address 127.0.0.1:58412)
0925 05:06:04.279213 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.279223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:04.279265 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:04.279548 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:04.279549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:04.279551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:04.279592 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.279593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.280909 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.280909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.281918 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.281920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.282649 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.282650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.282710 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.282713 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:04.282769 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:06:04.282795 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260925 05:06:04.282894 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.279297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.14.33.1:39143)
0925 05:06:04.279398 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.279400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.279408 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:04.279433 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.279434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.279443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.279505 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:04.279642 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.280080 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.281997 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.282567 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.282639 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.282652 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.282792 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.282833 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":53,"threads_started":1}
I20260925 05:06:04.283080 21460 tablet_service.cc:1467] Processing CreateTablet for tablet 6859cb70bb0140bb8f565efac6dfac84 (DEFAULT_TABLE table=test-table [id=6f67e299b7c247aebe1f2bf0e5d3bd1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:04.283201 21460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6859cb70bb0140bb8f565efac6dfac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:04.284140 21894 tablet_bootstrap.cc:492] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Bootstrap starting.
I20260925 05:06:04.284160 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 6859cb70bb0140bb8f565efac6dfac84 (DEFAULT_TABLE table=test-table [id=6f67e299b7c247aebe1f2bf0e5d3bd1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:04.284296 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6859cb70bb0140bb8f565efac6dfac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:04.284511 21894 tablet_bootstrap.cc:654] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:04.284720 21895 tablet_bootstrap.cc:492] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18: Bootstrap starting.
I20260925 05:06:04.285049 21895 tablet_bootstrap.cc:654] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:04.285142 21894 tablet_bootstrap.cc:492] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: No bootstrap required, opened a new log
I20260925 05:06:04.285211 21894 ts_tablet_manager.cc:1397] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:04.285331 21894 raft_consensus.cc:348] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.285429 21894 raft_consensus.cc:374] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:04.285455 21894 raft_consensus.cc:729] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9acf3f3d9af4931ba9623540c32ddfa, State: Initialized, Role: FOLLOWER
I20260925 05:06:04.285502 21894 consensus_queue.cc:260] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.285656 21894 ts_tablet_manager.cc:1428] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:04.285715 21895 tablet_bootstrap.cc:492] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18: No bootstrap required, opened a new log
I20260925 05:06:04.285807 21895 ts_tablet_manager.cc:1397] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:04.285941 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.285852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:51739)
0925 05:06:04.285889 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.285915 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 05:06:04.285974 21899 tablet_bootstrap.cc:492] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Bootstrap starting.
I20260925 05:06:04.285957 21895 raft_consensus.cc:348] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.286015 21895 raft_consensus.cc:374] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:04.286033 21895 raft_consensus.cc:729] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1d9d241f924f2384681a943df80a18, State: Initialized, Role: FOLLOWER
I20260925 05:06:04.286090 21895 consensus_queue.cc:260] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.286276 21895 ts_tablet_manager.cc:1428] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:04.286643 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.286508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:49569)
0925 05:06:04.286591 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.286618 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:06:04.286717 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.286630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:37711)
0925 05:06:04.286664 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.286687 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 05:06:04.286734 21899 tablet_bootstrap.cc:654] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:04.286787 21662 heartbeater.cc:503] Master 127.14.33.60:37515 was elected leader, sending a full tablet report...
I20260925 05:06:04.287165 21526 heartbeater.cc:503] Master 127.14.33.60:37515 was elected leader, sending a full tablet report...
I20260925 05:06:04.287446 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.287305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:51871)
0925 05:06:04.287391 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.287419 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 05:06:04.287529 21899 tablet_bootstrap.cc:492] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: No bootstrap required, opened a new log
I20260925 05:06:04.287570 21899 ts_tablet_manager.cc:1397] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:04.287720 21899 raft_consensus.cc:348] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.287807 21899 raft_consensus.cc:374] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:04.287856 21899 raft_consensus.cc:729] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294bef61bf664c5689c561d6e8bec574, State: Initialized, Role: FOLLOWER
I20260925 05:06:04.287904 21899 consensus_queue.cc:260] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.288064 21899 ts_tablet_manager.cc:1428] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:04.288329 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.288214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:49987)
0925 05:06:04.288276 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.288303 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 05:06:04.288430 21798 heartbeater.cc:503] Master 127.14.33.60:37515 was elected leader, sending a full tablet report...
I20260925 05:06:04.288590 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.288508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:57541)
0925 05:06:04.288544 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.288565 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 05:06:04.359292 21902 raft_consensus.cc:482] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:04.359450 21902 raft_consensus.cc:504] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.359807 21902 leader_election.cc:288] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747), 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143)
I20260925 05:06:04.363471 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.360051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:58043 (local address 127.14.33.1:39143)
0925 05:06:04.360112 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.360114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.360225 (+ 111us) server_negotiation.cc:407] Connection header received
0925 05:06:04.360272 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.360273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.360279 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.360338 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:04.360473 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.361046 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.362544 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.363033 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.363123 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.363137 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.363364 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.363399 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:06:04.363484 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.359926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39747 (local address 127.14.33.3:36401)
0925 05:06:04.359965 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.359976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:04.360026 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:04.360177 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:04.360178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:04.360181 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:04.360261 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.360263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.361057 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.361058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.362457 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.362459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.363157 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.363158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.363238 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.363241 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:04.363301 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:06:04.363344 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":70}
I20260925 05:06:04.363576 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.359994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:36401 (local address 127.14.33.2:39747)
0925 05:06:04.360031 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.360033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.360040 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:04.360086 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.360087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.360098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.360153 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:04.360331 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.360986 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.362544 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.363071 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.363442 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.363456 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.363475 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.363512 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:06:04.363750 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.359963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39143 (local address 127.14.33.3:58043)
0925 05:06:04.360150 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.360160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:04.360216 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:04.360360 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:04.360361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:04.360363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:04.360412 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.360413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.361108 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.361108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.362457 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:04.362459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.363193 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.363195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:04.363280 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.363283 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:04.363358 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:06:04.363396 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":136,"thread_start_us":97,"threads_started":1}
I20260925 05:06:04.363759 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6859cb70bb0140bb8f565efac6dfac84" candidate_uuid: "294bef61bf664c5689c561d6e8bec574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" is_pre_election: true
I20260925 05:06:04.363888 21616 raft_consensus.cc:2454] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294bef61bf664c5689c561d6e8bec574 in term 0.
I20260925 05:06:04.364008 21480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6859cb70bb0140bb8f565efac6dfac84" candidate_uuid: "294bef61bf664c5689c561d6e8bec574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c1d9d241f924f2384681a943df80a18" is_pre_election: true
I20260925 05:06:04.364042 21686 leader_election.cc:302] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294bef61bf664c5689c561d6e8bec574, b9acf3f3d9af4931ba9623540c32ddfa; no voters:
I20260925 05:06:04.364094 21480 raft_consensus.cc:2454] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294bef61bf664c5689c561d6e8bec574 in term 0.
I20260925 05:06:04.364168 21902 raft_consensus.cc:2781] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:04.364217 21902 raft_consensus.cc:482] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:04.364238 21902 raft_consensus.cc:3037] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:04.364845 21902 raft_consensus.cc:504] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.364976 21902 leader_election.cc:288] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [CANDIDATE]: Term 1 election: Requested vote from peers b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747), 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143)
I20260925 05:06:04.365099 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6859cb70bb0140bb8f565efac6dfac84" candidate_uuid: "294bef61bf664c5689c561d6e8bec574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9acf3f3d9af4931ba9623540c32ddfa"
I20260925 05:06:04.365175 21616 raft_consensus.cc:3037] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:04.365140 21480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6859cb70bb0140bb8f565efac6dfac84" candidate_uuid: "294bef61bf664c5689c561d6e8bec574" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c1d9d241f924f2384681a943df80a18"
I20260925 05:06:04.365208 21480 raft_consensus.cc:3037] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:04.365813 21480 raft_consensus.cc:2454] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294bef61bf664c5689c561d6e8bec574 in term 1.
I20260925 05:06:04.365859 21616 raft_consensus.cc:2454] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294bef61bf664c5689c561d6e8bec574 in term 1.
I20260925 05:06:04.365962 21685 leader_election.cc:302] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294bef61bf664c5689c561d6e8bec574, 2c1d9d241f924f2384681a943df80a18; no voters:
I20260925 05:06:04.366068 21902 raft_consensus.cc:2781] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:04.366189 21902 raft_consensus.cc:686] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 1 LEADER]: Becoming Leader. State: Replica: 294bef61bf664c5689c561d6e8bec574, State: Running, Role: LEADER
I20260925 05:06:04.366261 21902 consensus_queue.cc:237] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } }
I20260925 05:06:04.366760 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.366681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:58523)
0925 05:06:04.366709 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.366736 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 05:06:04.366909 21345 catalog_manager.cc:5495] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294bef61bf664c5689c561d6e8bec574 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "294bef61bf664c5689c561d6e8bec574" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294bef61bf664c5689c561d6e8bec574" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 43701 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:04.372260 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.370525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.14.33.3:43701)
0925 05:06:04.370555 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:04.370557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:04.370564 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:04.370614 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:04.370615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:04.370627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:04.370684 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:04.370756 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.371128 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.371732 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:04.371884 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:04.371926 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:04.372036 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:04.372115 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:04.372161 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:04.372206 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 05:06:04.372218 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:06:04.373049 21480 raft_consensus.cc:1261] T 6859cb70bb0140bb8f565efac6dfac84 P 2c1d9d241f924f2384681a943df80a18 [term 1 FOLLOWER]: Refusing update from remote peer 294bef61bf664c5689c561d6e8bec574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:04.373152 21616 raft_consensus.cc:1261] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Refusing update from remote peer 294bef61bf664c5689c561d6e8bec574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:04.373377 21904 consensus_queue.cc:1035] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c1d9d241f924f2384681a943df80a18" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:04.373528 21904 consensus_queue.cc:1035] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9acf3f3d9af4931ba9623540c32ddfa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:04.374354 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.374272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:53655)
0925 05:06:04.374301 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.374327 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 05:06:04.374502 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.374419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:32975)
0925 05:06:04.374453 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.374477 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 05:06:04.375481 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.375380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:33095)
0925 05:06:04.375435 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.375455 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:06:04.375667 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.375572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.2:49851)
0925 05:06:04.375604 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.375623 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 05:06:04.376317 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.376235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.3:51901)
0925 05:06:04.376268 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.376289 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 05:06:04.376547 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.376112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39543 (local address 127.14.33.1:58985)
0925 05:06:04.376496 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.376522 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370}
I20260925 05:06:04.376994 21910 mvcc.cc:205] Tried to move back new op lower bound from 7333121082870636544 to 7333121082844753920. Current Snapshot: MvccSnapshot[applied={T|T < 7333121082870636544 or (T in {7333121082870636544})}]
I20260925 05:06:04.377812 21345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:04.377892 21345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:04.378878 21345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19fe7dc2d89e41e0ba921f3524f590a7: Sending DeleteTablet for 3 replicas of tablet 6859cb70bb0140bb8f565efac6dfac84
I20260925 05:06:04.379104 21596 tablet_service.cc:1514] Processing DeleteTablet for tablet 6859cb70bb0140bb8f565efac6dfac84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:04 UTC) from {username='oryx', principal='oryx/127.14.33.60@KRBTEST.COM'} at 127.0.0.1:41064
I20260925 05:06:04.379151 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 6859cb70bb0140bb8f565efac6dfac84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:04 UTC) from {username='oryx', principal='oryx/127.14.33.60@KRBTEST.COM'} at 127.0.0.1:36876
I20260925 05:06:04.379308 21919 tablet_replica.cc:330] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: stopping tablet replica
I20260925 05:06:04.379310 21460 tablet_service.cc:1514] Processing DeleteTablet for tablet 6859cb70bb0140bb8f565efac6dfac84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:04 UTC) from {username='oryx', principal='oryx/127.14.33.60@KRBTEST.COM'} at 127.0.0.1:58412
I20260925 05:06:04.379391 21919 raft_consensus.cc:2229] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:04.379557 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 21397
I20260925 05:06:04.379485 21919 raft_consensus.cc:2258] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:04.380342 21919 ts_tablet_manager.cc:1905] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:04.380393 21918 tablet_replica.cc:330] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: stopping tablet replica
I20260925 05:06:04.380507 21918 raft_consensus.cc:2229] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:04.380602 21918 raft_consensus.cc:2258] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:04.380909 21918 ts_tablet_manager.cc:1905] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:04.381855 21919 ts_tablet_manager.cc:1918] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:04.381950 21919 log.cc:1196] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/6859cb70bb0140bb8f565efac6dfac84
I20260925 05:06:04.382282 21919 ts_tablet_manager.cc:1939] T 6859cb70bb0140bb8f565efac6dfac84 P 294bef61bf664c5689c561d6e8bec574: Deleting consensus metadata
I20260925 05:06:04.382601 21329 catalog_manager.cc:4841] TS 294bef61bf664c5689c561d6e8bec574 (127.14.33.3:43701): tablet 6859cb70bb0140bb8f565efac6dfac84 (table test-table [id=6f67e299b7c247aebe1f2bf0e5d3bd1a]) successfully deleted
I20260925 05:06:04.382722 21918 ts_tablet_manager.cc:1918] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:04.382782 21918 log.cc:1196] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/6859cb70bb0140bb8f565efac6dfac84
I20260925 05:06:04.383049 21918 ts_tablet_manager.cc:1939] T 6859cb70bb0140bb8f565efac6dfac84 P b9acf3f3d9af4931ba9623540c32ddfa: Deleting consensus metadata
I20260925 05:06:04.383330 21330 catalog_manager.cc:4841] TS b9acf3f3d9af4931ba9623540c32ddfa (127.14.33.2:39747): tablet 6859cb70bb0140bb8f565efac6dfac84 (table test-table [id=6f67e299b7c247aebe1f2bf0e5d3bd1a]) successfully deleted
I20260925 05:06:04.387084 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:04.386966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39143 (local address 127.0.0.1:58420)
0925 05:06:04.387024 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:04.387051 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:06:04.387097 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 21533
W20260925 05:06:04.387202 21329 catalog_manager.cc:4584] TS 2c1d9d241f924f2384681a943df80a18 (127.14.33.1:39143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6859cb70bb0140bb8f565efac6dfac84: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39143: connect: Connection refused (error 111)
I20260925 05:06:04.393360 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 21669
I20260925 05:06:04.398025 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 21025
I20260925 05:06:04.402204 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 21212
2026-09-25T05:06:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6235 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:04 dist-test-slave-4dmq krb5kdc[21923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:04 dist-test-slave-4dmq krb5kdc[21923](info): set up 2 sockets
Sep 25 05:06:04 dist-test-slave-4dmq krb5kdc[21923](info): commencing operation
krb5kdc: starting...
W20260925 05:06:05.374604 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab.
Entry for principal oryx/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab.
I20260925 05:06:05.480018 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:39063
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab
--principal=oryx/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:05.560423 21936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:05.560595 21936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:05.560616 21936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:05.561609 21936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:05.561645 21936 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 05:06:05.561659 21936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:05.561672 21936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:05.561684 21936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:05.561695 21936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:05.562970 21936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:39063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.62.keytab
--principal=oryx/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:05.563171 21936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:05.563342 21936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:05.568094 21942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:05.568092 21944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:05.568091 21941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:05.568123 21936 server_base.cc:1034] running on GCE node
I20260925 05:06:05.568663 21936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:05.568914 21936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:05.570061 21936 hybrid_clock.cc:648] HybridClock initialized: now 1790312765570054 us; error 33 us; skew 500 ppm
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:05.572280 21936 init.cc:377] Logged in from keytab as oryx/127.14.33.62@KRBTEST.COM (short username oryx)
I20260925 05:06:05.572679 21936 webserver.cc:460] Webserver started at http://127.14.33.62:39949/ using document root <none> and password file <none>
I20260925 05:06:05.572898 21936 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:05.572960 21936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:05.573081 21936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:05.574064 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "ee4c126c743947648d98e741f8ad143c"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "97027bcb0d6c960a348cc26779c35d90"
server_key_iv: "5e461d4ce4a01ea908999b00de2c7341"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.574399 21936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "ee4c126c743947648d98e741f8ad143c"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "97027bcb0d6c960a348cc26779c35d90"
server_key_iv: "5e461d4ce4a01ea908999b00de2c7341"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.575613 21936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:06:05.576236 21951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.576402 21936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:05.576491 21936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "ee4c126c743947648d98e741f8ad143c"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "97027bcb0d6c960a348cc26779c35d90"
server_key_iv: "5e461d4ce4a01ea908999b00de2c7341"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.576576 21936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:05.586553 21936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:05.588165 21936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:05.588315 21936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:05.592746 21936 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:39063
I20260925 05:06:05.592785 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:39063 every 8 connection(s)
I20260925 05:06:05.593075 21936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:05.593698 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:05.594147 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 21936
I20260925 05:06:05.594242 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:06:05.594424 14468 external_mini_cluster.cc:1428] Setting key bd2851e12746bc201ea6e84d53e977ba
I20260925 05:06:05.596367 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Bootstrap starting.
I20260925 05:06:05.596917 22014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:05.597288 22014 log.cc:824] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:05.597971 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: No bootstrap required, opened a new log
I20260925 05:06:05.600354 22014 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4c126c743947648d98e741f8ad143c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39063 } }
I20260925 05:06:05.600476 22014 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:05.600497 22014 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4c126c743947648d98e741f8ad143c, State: Initialized, Role: FOLLOWER
I20260925 05:06:05.600613 22014 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4c126c743947648d98e741f8ad143c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39063 } }
I20260925 05:06:05.600678 22014 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:05.600713 22014 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:05.600765 22014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:05.601274 22014 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4c126c743947648d98e741f8ad143c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39063 } }
I20260925 05:06:05.601378 22014 leader_election.cc:302] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee4c126c743947648d98e741f8ad143c; no voters:
I20260925 05:06:05.601519 22014 leader_election.cc:288] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:05.601567 22019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:05.601747 22019 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [term 1 LEADER]: Becoming Leader. State: Replica: ee4c126c743947648d98e741f8ad143c, State: Running, Role: LEADER
I20260925 05:06:05.601850 22014 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:05.601895 22019 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4c126c743947648d98e741f8ad143c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39063 } }
I20260925 05:06:05.602279 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4c126c743947648d98e741f8ad143c. Latest consensus state: current_term: 1 leader_uuid: "ee4c126c743947648d98e741f8ad143c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4c126c743947648d98e741f8ad143c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39063 } } }
I20260925 05:06:05.602265 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee4c126c743947648d98e741f8ad143c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4c126c743947648d98e741f8ad143c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 39063 } } }
I20260925 05:06:05.602473 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:05.602511 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:05.602659 22028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:05.603125 22028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:05.604686 22028 catalog_manager.cc:1347] Generated new cluster ID: 83cfb116fa6347c48fa27c2a628b36f2
I20260925 05:06:05.604730 22028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:05.604745 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.594872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.14.33.62:39063)
0925 05:06:05.595032 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:05.595039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:05.595066 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:06:05.595397 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:05.595402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:05.595440 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:05.595539 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:05.596240 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.597149 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.598132 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.598501 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.598575 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.601185 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:05.601189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:05.601189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:05.601199 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:05.603112 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.603429 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.603432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.603432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.603554 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.603766 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.603766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.603767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.604002 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:05.604142 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:05.604447 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 05:06:05.604574 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 05:06:05.634913 22028 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:05.635567 22028 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:05.644307 22028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Generated new TSK 0
I20260925 05:06:05.644469 22028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab.
Entry for principal oryx/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab.
I20260925 05:06:05.673839 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39063
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:05.748189 22040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:05.748365 22040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:05.748387 22040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:05.749476 22040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:05.749531 22040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:05.749550 22040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:05.749594 22040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:05.750996 22040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:05.751194 22040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:05.751412 22040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:05.755888 22046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:05.755915 22048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:05.755934 22040 server_base.cc:1034] running on GCE node
W20260925 05:06:05.755903 22045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:05.756283 22040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:05.756450 22040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:05.757572 22040 hybrid_clock.cc:648] HybridClock initialized: now 1790312765757564 us; error 24 us; skew 500 ppm
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:05.759856 22040 init.cc:377] Logged in from keytab as oryx/127.14.33.1@KRBTEST.COM (short username oryx)
I20260925 05:06:05.760254 22040 webserver.cc:460] Webserver started at http://127.14.33.1:38381/ using document root <none> and password file <none>
I20260925 05:06:05.760432 22040 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:05.760468 22040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:05.760550 22040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:05.761466 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "15d71d75eb7c490daefff6b693cb181c"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "9efd01f598c42bbd4cc32dca7e12655d"
server_key_iv: "b336a141dc6984878b589e878908d6ff"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.761740 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "15d71d75eb7c490daefff6b693cb181c"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "9efd01f598c42bbd4cc32dca7e12655d"
server_key_iv: "b336a141dc6984878b589e878908d6ff"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.762853 22040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:05.763458 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.763620 22040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:05.763692 22040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "15d71d75eb7c490daefff6b693cb181c"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "9efd01f598c42bbd4cc32dca7e12655d"
server_key_iv: "b336a141dc6984878b589e878908d6ff"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.763741 22040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:05.772979 22040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:05.774930 22040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:05.775024 22040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:05.775403 22040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:05.775465 22040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.775506 22040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:05.775528 22040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.781680 22040 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:45663
I20260925 05:06:05.781760 22170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:45663 every 8 connection(s)
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:05.782075 22040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:05.784806 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.775842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:43079 (local address 127.14.33.62:39063)
0925 05:06:05.775904 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:05.775907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:05.776083 (+ 176us) server_negotiation.cc:407] Connection header received
0925 05:06:05.776493 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:05.776495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:05.776508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:05.776601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:05.777356 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.778186 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.779660 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.780025 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.780092 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.782543 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:05.782545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:05.782545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:05.782552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:05.783791 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.784105 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.784106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.784106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.784152 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.784350 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.784351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.784351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.784442 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:05.784490 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:05.784676 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 05:06:05.784735 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:06:05.784991 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.775816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.14.33.1:43079)
0925 05:06:05.775964 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:05.775998 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:05.776410 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:05.776645 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:05.776648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:05.776955 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:05.777245 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.777251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.778278 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.778278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.779582 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.779585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.780101 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.780102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.780200 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.780640 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:05.780652 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:05.782444 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:05.783853 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.783857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.783858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.784049 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.784191 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.784191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.784192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.784298 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.784493 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:05.784497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:05.784593 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 05:06:05.784666 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:06:05.784768 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260925 05:06:05.785480 22171 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39063
I20260925 05:06:05.785557 22171 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:05.785724 22171 heartbeater.cc:511] Master 127.14.33.62:39063 requested a full tablet report, sending...
I20260925 05:06:05.786173 21967 ts_manager.cc:194] Registered new tserver with Master: 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663)
I20260925 05:06:05.786931 21967 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:43079
I20260925 05:06:05.787611 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22040
I20260925 05:06:05.787717 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:05.787904 14468 external_mini_cluster.cc:1428] Setting key b4d72bdfb2ee019766e907e054384f77
I20260925 05:06:05.791499 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.786420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:41197 (local address 127.14.33.62:39063)
0925 05:06:05.786451 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:05.786453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:05.786524 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:06:05.786584 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:05.786584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:05.786592 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:05.786640 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:05.786868 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.787402 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.788223 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.788500 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.788557 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.789332 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:05.789333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:05.789334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:05.789340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:05.790507 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.790808 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.790809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.790810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.790862 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.791032 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.791032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.791033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.791144 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:05.791211 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:05.791349 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 05:06:05.791415 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:06:05.791505 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.786367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.14.33.1:41197)
0925 05:06:05.786477 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:05.786486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:05.786537 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:05.786643 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:05.786643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:05.786769 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:05.786809 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.786817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.787485 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.787486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.788166 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.788168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.788575 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.788576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.788632 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.788976 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:05.788979 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:05.789255 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:05.790596 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.790597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.790597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.790746 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.790918 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.790919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.790920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.790970 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.791204 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:05.791204 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:05.791245 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:06:05.791324 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:06:05.791417 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab.
Entry for principal oryx/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab.
I20260925 05:06:05.811272 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39063
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:05.885161 22180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:05.885380 22180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:05.885416 22180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:05.886534 22180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:05.886590 22180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:05.886622 22180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:05.886669 22180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:05.888100 22180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:05.888347 22180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:05.888547 22180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:05.893287 22188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:05.893356 22186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:05.893388 22180 server_base.cc:1034] running on GCE node
W20260925 05:06:05.893352 22185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:05.893729 22180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:05.893996 22180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:05.895156 22180 hybrid_clock.cc:648] HybridClock initialized: now 1790312765895135 us; error 34 us; skew 500 ppm
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:05.897383 22180 init.cc:377] Logged in from keytab as oryx/127.14.33.2@KRBTEST.COM (short username oryx)
I20260925 05:06:05.897781 22180 webserver.cc:460] Webserver started at http://127.14.33.2:41151/ using document root <none> and password file <none>
I20260925 05:06:05.897987 22180 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:05.898047 22180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:05.898182 22180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:05.899109 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "6d67995f08d140ceaa09a80f469903e0"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "8accc28f6e7e1d726efe4d2583a302ac"
server_key_iv: "6f334342de60d4cba61d28ebc5149fdd"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.899428 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "6d67995f08d140ceaa09a80f469903e0"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "8accc28f6e7e1d726efe4d2583a302ac"
server_key_iv: "6f334342de60d4cba61d28ebc5149fdd"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.900684 22180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:06:05.901321 22195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.901471 22180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.901556 22180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "6d67995f08d140ceaa09a80f469903e0"
format_stamp: "Formatted at 2026-09-25 05:06:05 on dist-test-slave-4dmq"
server_key: "8accc28f6e7e1d726efe4d2583a302ac"
server_key_iv: "6f334342de60d4cba61d28ebc5149fdd"
server_key_version: "encryptionkey@0"
I20260925 05:06:05.901628 22180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:05.939613 22180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:05.941161 22180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:05.941296 22180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:05.941773 22180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:05.941811 22180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:05.941833 22180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:05.941882 22180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:06:05 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:05.948390 22180 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:44649
I20260925 05:06:05.948468 22310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:44649 every 8 connection(s)
I20260925 05:06:05.948808 22180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:05.951704 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.942342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:56739 (local address 127.14.33.62:39063)
0925 05:06:05.942399 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:05.942401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:05.942637 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:06:05.943137 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:05.943139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:05.943162 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:05.943228 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:05.944016 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.944801 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.945902 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.946225 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.946304 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.948716 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:05.948717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:05.948718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:05.948725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:05.950252 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.950629 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.950630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.950631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.950739 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.951033 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.951034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.951034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.951195 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:05.951293 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:05.951547 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 05:06:05.951622 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:06:05.951860 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.942318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.14.33.2:56739)
0925 05:06:05.942482 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:05.942534 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:05.943006 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:05.943302 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:05.943306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:05.943640 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:05.943915 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.943920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.944911 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.944912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.945831 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.945834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.946333 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.946334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.946472 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.946976 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:05.946986 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:05.948577 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:05.950324 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.950329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.950330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.950548 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.950821 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.950822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.950822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.950952 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.951287 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:05.951292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:05.951424 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 05:06:05.951520 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:06:05.951657 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260925 05:06:05.952320 22311 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39063
I20260925 05:06:05.952402 22311 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:05.952596 22311 heartbeater.cc:511] Master 127.14.33.62:39063 requested a full tablet report, sending...
I20260925 05:06:05.952963 21968 ts_manager.cc:194] Registered new tserver with Master: 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649)
I20260925 05:06:05.953471 21968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:56739
I20260925 05:06:05.955400 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22180
I20260925 05:06:05.955529 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:05.955724 14468 external_mini_cluster.cc:1428] Setting key a0e6e8a54454375844d4670fa9892886
I20260925 05:06:05.958930 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.953335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:40043 (local address 127.14.33.62:39063)
0925 05:06:05.953376 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:05.953377 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:05.953489 (+ 112us) server_negotiation.cc:407] Connection header received
0925 05:06:05.953546 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:05.953547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:05.953556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:05.953600 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:05.953924 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.954510 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.955136 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.955299 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.955352 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.956327 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:05.956327 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:05.956328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:05.956333 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:05.957719 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.958084 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.958086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.958086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.958144 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:05.958342 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:05.958343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:05.958344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:05.958521 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:05.958622 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:05.958768 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:06:05.958841 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:06:05.958945 22317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:05.953286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.14.33.2:40043)
0925 05:06:05.953423 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:05.953432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:05.953500 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:05.953638 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:05.953639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:05.953791 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:05.953847 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.953848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.954599 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.954599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.955069 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:05.955070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:05.955375 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:05.955375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:05.955437 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:05.955813 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:05.955818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:05.956192 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:05.957824 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.957825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.957825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.957998 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.958211 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:05.958212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:05.958212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:05.958274 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:05.958596 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:05.958597 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:05.958655 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 05:06:05.958744 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:06:05.958842 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab.
Entry for principal oryx/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab.
I20260925 05:06:05.979595 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:39063
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:06.055274 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:06.055445 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:06.055466 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:06.056531 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:06.056566 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:06.056586 22320 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 05:06:06.056603 22320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:06.058048 22320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.33.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:06.058210 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:06.058363 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:06.062942 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:06.062943 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:06.062965 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:06.063061 22320 server_base.cc:1034] running on GCE node
I20260925 05:06:06.063357 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:06.063549 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:06.064688 22320 hybrid_clock.cc:648] HybridClock initialized: now 1790312766064667 us; error 28 us; skew 500 ppm
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312766, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:06.067073 22320 init.cc:377] Logged in from keytab as oryx/127.14.33.3@KRBTEST.COM (short username oryx)
I20260925 05:06:06.067575 22320 webserver.cc:460] Webserver started at http://127.14.33.3:44469/ using document root <none> and password file <none>
I20260925 05:06:06.067786 22320 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:06.067828 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:06.067953 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:06.068961 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "e35bef191b3d4508bd71eb00c579f08c"
format_stamp: "Formatted at 2026-09-25 05:06:06 on dist-test-slave-4dmq"
server_key: "ef894504a28c7b66d968fbd6f2d2e795"
server_key_iv: "3308e3256a73875b460e276738e27bd5"
server_key_version: "encryptionkey@0"
I20260925 05:06:06.069332 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "e35bef191b3d4508bd71eb00c579f08c"
format_stamp: "Formatted at 2026-09-25 05:06:06 on dist-test-slave-4dmq"
server_key: "ef894504a28c7b66d968fbd6f2d2e795"
server_key_iv: "3308e3256a73875b460e276738e27bd5"
server_key_version: "encryptionkey@0"
I20260925 05:06:06.070545 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:06.071177 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.071327 22320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:06.071414 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "e35bef191b3d4508bd71eb00c579f08c"
format_stamp: "Formatted at 2026-09-25 05:06:06 on dist-test-slave-4dmq"
server_key: "ef894504a28c7b66d968fbd6f2d2e795"
server_key_iv: "3308e3256a73875b460e276738e27bd5"
server_key_version: "encryptionkey@0"
I20260925 05:06:06.071485 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:06.083528 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:06.085160 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:06.085345 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:06.085876 22320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:06.085928 22320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.085965 22320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:06.085987 22320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312766, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:06.092648 22320 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:39721
I20260925 05:06:06.092716 22450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:39721 every 8 connection(s)
I20260925 05:06:06.093014 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:06.093616 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22320
I20260925 05:06:06.093739 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:06.093945 14468 external_mini_cluster.cc:1428] Setting key c5a36f2e88a6514cf342d1fcd8f8cdbf
I20260925 05:06:06.095520 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.086210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:34221 (local address 127.14.33.62:39063)
0925 05:06:06.086260 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.086262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.086443 (+ 181us) server_negotiation.cc:407] Connection header received
0925 05:06:06.086917 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.086918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.086931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.087004 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.087569 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.088386 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.089644 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.089848 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.089918 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.092512 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:06.092513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:06.092513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:06.092521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:06.094226 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.094584 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.094584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.094585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.094630 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.094882 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.094883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.094884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.094991 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:06.095059 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.095315 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.095433 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:06:06.095660 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.086200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.14.33.3:34221)
0925 05:06:06.086326 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.086361 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.086829 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.087049 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.087052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.087274 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:06.087459 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.087464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.088476 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.088477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.089557 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.089559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.089945 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.089947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.090105 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.090632 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:06.090640 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:06.092409 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:06.094292 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:06.094298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:06.094300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:06.094525 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:06.094704 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:06.094705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:06.094705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:06.094811 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:06.095064 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:06.095069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:06.095190 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.095291 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.095455 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260925 05:06:06.096035 22451 heartbeater.cc:347] Connected to a master server at 127.14.33.62:39063
I20260925 05:06:06.096108 22451 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:06.096283 22451 heartbeater.cc:511] Master 127.14.33.62:39063 requested a full tablet report, sending...
I20260925 05:06:06.096644 21968 ts_manager.cc:194] Registered new tserver with Master: e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.096987 21968 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:34221
I20260925 05:06:06.101471 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.097046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:59539 (local address 127.14.33.62:39063)
0925 05:06:06.097084 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.097085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.097247 (+ 162us) server_negotiation.cc:407] Connection header received
0925 05:06:06.097304 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.097304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.097314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.097363 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.097681 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.098153 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.098741 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.098905 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.098949 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.099622 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:06.099623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:06.099623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:06.099629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:06.100664 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.100935 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.100936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.100936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.100992 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.101095 (+ 103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.101095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.101095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.101221 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:06.101267 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.101372 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.101420 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:06:06.101486 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.096991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.14.33.3:59539)
0925 05:06:06.097185 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.097196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.097246 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.097367 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.097367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.097559 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:06.097617 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.097618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.098229 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.098230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.098674 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.098675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.098959 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.098960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.099000 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.099292 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:06.099295 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:06.099560 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:06.100727 (+ 1167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:06.100728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:06.100729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:06.100878 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:06.101028 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:06.101028 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:06.101028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:06.101065 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:06.101270 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:06.101271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:06.101302 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.101356 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.101423 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":61,"threads_started":1}
I20260925 05:06:06.105722 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:06:06.125519 22017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:06.120322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.14.33.62:39063)
0925 05:06:06.120370 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.120371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.120409 (+ 38us) server_negotiation.cc:407] Connection header received
0925 05:06:06.120472 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.120472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.120480 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.120526 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.121052 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.121695 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.122474 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.122703 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.122779 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.125442 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47922: BlockingRecv error: recv got EOF from 127.0.0.1:47922 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312766, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:06.131560 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.126158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.14.33.62:39063)
0925 05:06:06.126190 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.126192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.126293 (+ 101us) server_negotiation.cc:407] Connection header received
0925 05:06:06.126327 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.126328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.126335 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.126377 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.126769 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.127170 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.127740 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.127897 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.127945 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.129707 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:06.129709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:06.129709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:06.129714 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:06.130766 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.131045 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.131046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.131046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.131091 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.131226 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.131226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.131227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.131304 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:06.131367 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.131439 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.131488 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:06:06.132787 21968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:06.134339 21968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:06.140265 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.136451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.14.33.3:39721)
0925 05:06:06.136608 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.136613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.136877 (+ 264us) server_negotiation.cc:407] Connection header received
0925 05:06:06.136915 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.136918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.136934 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.137011 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.137179 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.138223 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.139429 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.139886 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.139966 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.140037 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.140136 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.140183 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260925 05:06:06.140259 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.136384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:39721 (local address 127.0.0.1:48612)
0925 05:06:06.136704 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.136759 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.136819 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.137014 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.137017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.137020 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.137076 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.137078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.138303 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.138304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.139371 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.139372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.139967 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.139968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.140042 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.140055 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.140124 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.140190 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":34,"thread_start_us":54,"threads_started":1}
I20260925 05:06:06.140386 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.136575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44649 (local address 127.0.0.1:46796)
0925 05:06:06.136764 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.136776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.136819 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.137186 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.137187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.137189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.137232 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.137233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.138071 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.138072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.139294 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.139296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.140111 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.140113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.140195 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.140203 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.140230 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.140286 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":45,"thread_start_us":60,"threads_started":1}
I20260925 05:06:06.140509 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.136384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45663 (local address 127.0.0.1:47402)
0925 05:06:06.136696 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.136734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.136797 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.137155 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.137155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.137157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.137197 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.137198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.138355 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.138356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.139155 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.139158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.140113 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.140114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.140195 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.140201 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.140245 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.140285 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":23,"thread_start_us":50,"threads_started":1}
I20260925 05:06:06.140790 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.136835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.14.33.2:44649)
0925 05:06:06.136982 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.136987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.136996 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:06.137028 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.137030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.137043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.137304 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.137358 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.137977 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.139373 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.140020 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.140496 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.140646 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.140667 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.140703 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260925 05:06:06.140789 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.136549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.14.33.1:45663)
0925 05:06:06.136935 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.136940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.136950 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:06:06.136986 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.136988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.137003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.137073 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.137283 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.138291 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.139241 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.139950 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.140017 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.140085 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.140652 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.140688 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":104,"threads_started":1}
I20260925 05:06:06.140892 22385 tablet_service.cc:1467] Processing CreateTablet for tablet 013c7c3a8cbb485e9925aefeb2603017 (DEFAULT_TABLE table=test-table [id=9683d6e0ec0e47138a48f01a043777ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.141292 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013c7c3a8cbb485e9925aefeb2603017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.141412 22245 tablet_service.cc:1467] Processing CreateTablet for tablet 013c7c3a8cbb485e9925aefeb2603017 (DEFAULT_TABLE table=test-table [id=9683d6e0ec0e47138a48f01a043777ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.141462 22105 tablet_service.cc:1467] Processing CreateTablet for tablet 013c7c3a8cbb485e9925aefeb2603017 (DEFAULT_TABLE table=test-table [id=9683d6e0ec0e47138a48f01a043777ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.141722 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013c7c3a8cbb485e9925aefeb2603017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.141722 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013c7c3a8cbb485e9925aefeb2603017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.143859 22476 tablet_bootstrap.cc:492] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Bootstrap starting.
I20260925 05:06:06.144400 22476 tablet_bootstrap.cc:654] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.144666 22476 log.cc:824] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:06.145248 22478 tablet_bootstrap.cc:492] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Bootstrap starting.
I20260925 05:06:06.145342 22476 tablet_bootstrap.cc:492] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: No bootstrap required, opened a new log
I20260925 05:06:06.145403 22476 ts_tablet_manager.cc:1397] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:06.145936 22478 tablet_bootstrap.cc:654] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.146238 22478 log.cc:824] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:06.146883 22478 tablet_bootstrap.cc:492] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: No bootstrap required, opened a new log
I20260925 05:06:06.146944 22478 ts_tablet_manager.cc:1397] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:06.147303 22476 raft_consensus.cc:348] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.147457 22476 raft_consensus.cc:374] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.147497 22476 raft_consensus.cc:729] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.147606 22476 consensus_queue.cc:260] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.147855 22476 ts_tablet_manager.cc:1428] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:06:06.147884 22451 heartbeater.cc:503] Master 127.14.33.62:39063 was elected leader, sending a full tablet report...
I20260925 05:06:06.148026 22479 tablet_bootstrap.cc:492] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Bootstrap starting.
I20260925 05:06:06.148576 22479 tablet_bootstrap.cc:654] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.148883 22479 log.cc:824] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:06.148885 22478 raft_consensus.cc:348] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.149022 22478 raft_consensus.cc:374] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.149068 22478 raft_consensus.cc:729] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.149195 22478 consensus_queue.cc:260] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.149432 22478 ts_tablet_manager.cc:1428] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:06:06.149458 22311 heartbeater.cc:503] Master 127.14.33.62:39063 was elected leader, sending a full tablet report...
I20260925 05:06:06.149605 22479 tablet_bootstrap.cc:492] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: No bootstrap required, opened a new log
I20260925 05:06:06.149672 22479 ts_tablet_manager.cc:1397] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:06.151115 22479 raft_consensus.cc:348] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.151229 22479 raft_consensus.cc:374] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.151250 22479 raft_consensus.cc:729] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d71d75eb7c490daefff6b693cb181c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.151350 22479 consensus_queue.cc:260] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.151620 22479 ts_tablet_manager.cc:1428] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:06:06.151697 22171 heartbeater.cc:503] Master 127.14.33.62:39063 was elected leader, sending a full tablet report...
W20260925 05:06:06.199432 22312 tablet.cc:2369] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:06.249670 22483 raft_consensus.cc:482] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:06.249776 22483 raft_consensus.cc:504] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.250115 22483 leader_election.cc:288] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.251104 22481 raft_consensus.cc:482] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:06.251214 22481 raft_consensus.cc:504] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.251811 22481 leader_election.cc:288] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649)
I20260925 05:06:06.253880 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.250358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:58127 (local address 127.14.33.3:39721)
0925 05:06:06.250416 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.250418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.250536 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:06:06.250567 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.250568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.250591 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.250669 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.250840 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.251426 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.252969 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253531 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253570 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.253581 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.253780 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.253813 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:06:06.253940 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.250276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45663 (local address 127.14.33.2:52745)
0925 05:06:06.250332 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.250343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.250390 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.250566 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.250567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.250569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.250671 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.250673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.251476 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.251477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.252917 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.252919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253680 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.253774 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.253777 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.253821 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.253857 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":237}
I20260925 05:06:06.254249 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.250337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:52745 (local address 127.14.33.1:45663)
0925 05:06:06.250380 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.250382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.250391 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:06.250482 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.250483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.250495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.250549 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.250743 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.251397 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253014 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253542 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253605 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.253615 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.254142 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.254184 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:06:06.254516 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013c7c3a8cbb485e9925aefeb2603017" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c" is_pre_election: true
I20260925 05:06:06.254666 22125 raft_consensus.cc:2454] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 0.
I20260925 05:06:06.254874 22199 leader_election.cc:302] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, 6d67995f08d140ceaa09a80f469903e0; no voters:
I20260925 05:06:06.255019 22483 raft_consensus.cc:2781] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:06.255075 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.251638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45663 (local address 127.14.33.3:48543)
0925 05:06:06.251739 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.251764 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.251808 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.252052 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.252053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.252055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.252109 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.252110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253023 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.254111 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.254114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.254793 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.254794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.254862 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.254865 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.254963 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.255006 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 05:06:06.255085 22483 raft_consensus.cc:482] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:06.255131 22483 raft_consensus.cc:3037] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.255213 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.250289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:39721 (local address 127.14.33.2:58127)
0925 05:06:06.250459 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.250469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.250522 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.250684 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.250684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.250685 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.250773 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.250775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.251516 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.251517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.252870 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.252873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253633 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.253702 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.253705 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.254372 (+ 667us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.255036 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260925 05:06:06.255766 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013c7c3a8cbb485e9925aefeb2603017" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35bef191b3d4508bd71eb00c579f08c" is_pre_election: true
I20260925 05:06:06.255849 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.251882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:36287 (local address 127.14.33.2:44649)
0925 05:06:06.251922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.251923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.252028 (+ 105us) server_negotiation.cc:407] Connection header received
0925 05:06:06.252117 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.252118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.252126 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.252154 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.252367 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253430 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.254985 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.255493 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.255575 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.255589 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.255731 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.255778 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:06:06.255894 22402 raft_consensus.cc:2454] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 0.
I20260925 05:06:06.255919 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.251732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:48543 (local address 127.14.33.1:45663)
0925 05:06:06.251888 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.251890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.251934 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:06:06.251955 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.251956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.251964 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.252168 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.252197 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.252936 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.254216 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.254729 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.254793 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.254802 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.254970 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.255840 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260925 05:06:06.256042 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.251674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44649 (local address 127.14.33.3:36287)
0925 05:06:06.251966 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.251976 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.252051 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.252203 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.252204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.252205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.252265 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.252266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.253536 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.253537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.254876 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.254879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.255596 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.255597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.255664 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.255668 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.255726 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.255766 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":85,"threads_started":1}
I20260925 05:06:06.256069 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013c7c3a8cbb485e9925aefeb2603017" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c" is_pre_election: true
I20260925 05:06:06.256135 22125 raft_consensus.cc:2454] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 0.
I20260925 05:06:06.256134 22483 raft_consensus.cc:504] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.256271 22483 leader_election.cc:288] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 election: Requested vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.256306 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013c7c3a8cbb485e9925aefeb2603017" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0" is_pre_election: true
I20260925 05:06:06.256398 22265 raft_consensus.cc:2379] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e35bef191b3d4508bd71eb00c579f08c in current term 1: Already voted for candidate 6d67995f08d140ceaa09a80f469903e0 in this term.
I20260925 05:06:06.256444 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013c7c3a8cbb485e9925aefeb2603017" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c"
I20260925 05:06:06.256496 22125 raft_consensus.cc:3037] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.256608 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013c7c3a8cbb485e9925aefeb2603017" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35bef191b3d4508bd71eb00c579f08c"
I20260925 05:06:06.256641 22339 leader_election.cc:302] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.256677 22402 raft_consensus.cc:3037] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.257418 22402 raft_consensus.cc:2454] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 1.
I20260925 05:06:06.257498 22481 raft_consensus.cc:2741] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 05:06:06.257591 22125 raft_consensus.cc:2454] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 1.
I20260925 05:06:06.257627 22196 leader_election.cc:302] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d67995f08d140ceaa09a80f469903e0, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.257712 22483 raft_consensus.cc:2781] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:06.257833 22483 raft_consensus.cc:686] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Running, Role: LEADER
I20260925 05:06:06.257918 22483 consensus_queue.cc:237] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } }
I20260925 05:06:06.258674 21968 catalog_manager.cc:5495] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2). New cstate: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } health_report { overall_health: HEALTHY } } }
W20260925 05:06:06.282914 22172 tablet.cc:2369] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:06.287288 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.284880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.14.33.2:44649)
0925 05:06:06.284937 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.284939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.284948 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:06.285040 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.285041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.285064 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.285155 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:06.285281 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.285705 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.286537 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.286728 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.286801 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.286963 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.287067 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.287156 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.287185 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.287205 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:06:06.289045 22125 raft_consensus.cc:1261] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Refusing update from remote peer 6d67995f08d140ceaa09a80f469903e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.289119 22402 raft_consensus.cc:1261] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Refusing update from remote peer 6d67995f08d140ceaa09a80f469903e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.289472 22488 consensus_queue.cc:1035] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.289618 22483 consensus_queue.cc:1035] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.292466 22493 mvcc.cc:205] Tried to move back new op lower bound from 7333121090717163520 to 7333121090592915456. Current Snapshot: MvccSnapshot[applied={T|T < 7333121090717163520}]
I20260925 05:06:06.292891 22492 mvcc.cc:205] Tried to move back new op lower bound from 7333121090717163520 to 7333121090592915456. Current Snapshot: MvccSnapshot[applied={T|T < 7333121090717163520}]
I20260925 05:06:06.294275 21968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:06.294346 21968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:06.295050 21968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Sending DeleteTablet for 3 replicas of tablet 013c7c3a8cbb485e9925aefeb2603017
I20260925 05:06:06.295428 22105 tablet_service.cc:1514] Processing DeleteTablet for tablet 013c7c3a8cbb485e9925aefeb2603017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:47402
I20260925 05:06:06.295456 22245 tablet_service.cc:1514] Processing DeleteTablet for tablet 013c7c3a8cbb485e9925aefeb2603017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46796
I20260925 05:06:06.295472 22385 tablet_service.cc:1514] Processing DeleteTablet for tablet 013c7c3a8cbb485e9925aefeb2603017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:48612
I20260925 05:06:06.295624 22501 tablet_replica.cc:330] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: stopping tablet replica
I20260925 05:06:06.295711 22501 raft_consensus.cc:2229] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:06.295804 22501 raft_consensus.cc:2258] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.295905 21968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:06:06.295974 22502 tablet_replica.cc:330] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: stopping tablet replica
W20260925 05:06:06.296056 21968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:06.296069 22502 raft_consensus.cc:2229] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:06.296147 22502 raft_consensus.cc:2258] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.296510 22500 tablet_replica.cc:330] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: stopping tablet replica
I20260925 05:06:06.296530 22501 ts_tablet_manager.cc:1905] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.296593 22500 raft_consensus.cc:2229] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:06.296689 22500 raft_consensus.cc:2258] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.297606 22245 tablet_service.cc:1467] Processing CreateTablet for tablet 624960bf89464d74a5284f6eb1d35d16 (DEFAULT_TABLE table=test-table [id=842ca062cab047abad9473e069d12768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.297611 22385 tablet_service.cc:1467] Processing CreateTablet for tablet 624960bf89464d74a5284f6eb1d35d16 (DEFAULT_TABLE table=test-table [id=842ca062cab047abad9473e069d12768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.297703 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624960bf89464d74a5284f6eb1d35d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.297734 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624960bf89464d74a5284f6eb1d35d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.298450 22105 tablet_service.cc:1467] Processing CreateTablet for tablet 624960bf89464d74a5284f6eb1d35d16 (DEFAULT_TABLE table=test-table [id=842ca062cab047abad9473e069d12768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.298548 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624960bf89464d74a5284f6eb1d35d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.299057 22476 tablet_bootstrap.cc:492] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Bootstrap starting.
I20260925 05:06:06.299412 22501 ts_tablet_manager.cc:1918] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:06.299427 22476 tablet_bootstrap.cc:654] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.299467 22501 log.cc:1196] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/013c7c3a8cbb485e9925aefeb2603017
I20260925 05:06:06.299700 22501 ts_tablet_manager.cc:1939] T 013c7c3a8cbb485e9925aefeb2603017 P 6d67995f08d140ceaa09a80f469903e0: Deleting consensus metadata
I20260925 05:06:06.299717 22478 tablet_bootstrap.cc:492] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Bootstrap starting.
I20260925 05:06:06.299888 22479 tablet_bootstrap.cc:492] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Bootstrap starting.
I20260925 05:06:06.300089 21952 catalog_manager.cc:4841] TS 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649): tablet 013c7c3a8cbb485e9925aefeb2603017 (table test-table [id=9683d6e0ec0e47138a48f01a043777ca]) successfully deleted
I20260925 05:06:06.300114 22478 tablet_bootstrap.cc:654] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.300295 22479 tablet_bootstrap.cc:654] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.300382 22500 ts_tablet_manager.cc:1905] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.301028 22478 tablet_bootstrap.cc:492] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: No bootstrap required, opened a new log
I20260925 05:06:06.301093 22478 ts_tablet_manager.cc:1397] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:06.301203 22479 tablet_bootstrap.cc:492] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: No bootstrap required, opened a new log
I20260925 05:06:06.301257 22479 ts_tablet_manager.cc:1397] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:06.301257 22478 raft_consensus.cc:348] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.301321 22478 raft_consensus.cc:374] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.301342 22478 raft_consensus.cc:729] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.301384 22478 consensus_queue.cc:260] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.301393 22479 raft_consensus.cc:348] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.301461 22478 ts_tablet_manager.cc:1428] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.301467 22479 raft_consensus.cc:374] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.301492 22479 raft_consensus.cc:729] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d71d75eb7c490daefff6b693cb181c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.301566 22479 consensus_queue.cc:260] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.301653 22479 ts_tablet_manager.cc:1428] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.301937 22502 ts_tablet_manager.cc:1905] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.302589 22500 ts_tablet_manager.cc:1918] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:06.302683 22500 log.cc:1196] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/013c7c3a8cbb485e9925aefeb2603017
I20260925 05:06:06.302939 22500 ts_tablet_manager.cc:1939] T 013c7c3a8cbb485e9925aefeb2603017 P 15d71d75eb7c490daefff6b693cb181c: Deleting consensus metadata
I20260925 05:06:06.303288 21955 catalog_manager.cc:4841] TS 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663): tablet 013c7c3a8cbb485e9925aefeb2603017 (table test-table [id=9683d6e0ec0e47138a48f01a043777ca]) successfully deleted
I20260925 05:06:06.303654 22502 ts_tablet_manager.cc:1918] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:06.303732 22502 log.cc:1196] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/013c7c3a8cbb485e9925aefeb2603017
I20260925 05:06:06.304018 22502 ts_tablet_manager.cc:1939] T 013c7c3a8cbb485e9925aefeb2603017 P e35bef191b3d4508bd71eb00c579f08c: Deleting consensus metadata
I20260925 05:06:06.304378 21952 catalog_manager.cc:4841] TS e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721): tablet 013c7c3a8cbb485e9925aefeb2603017 (table test-table [id=9683d6e0ec0e47138a48f01a043777ca]) successfully deleted
I20260925 05:06:06.304606 22476 tablet_bootstrap.cc:492] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: No bootstrap required, opened a new log
I20260925 05:06:06.304682 22476 ts_tablet_manager.cc:1397] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260925 05:06:06.304823 22476 raft_consensus.cc:348] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.304889 22476 raft_consensus.cc:374] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.304912 22476 raft_consensus.cc:729] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.304963 22476 consensus_queue.cc:260] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.305056 22476 ts_tablet_manager.cc:1428] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260925 05:06:06.343796 22452 tablet.cc:2369] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:06.375165 22484 raft_consensus.cc:482] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:06.375247 22484 raft_consensus.cc:504] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.375548 22484 leader_election.cc:288] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649), e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.379137 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.376022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:59061 (local address 127.14.33.2:44649)
0925 05:06:06.376087 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.376089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.376099 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:06:06.376127 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.376128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.376137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.376218 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.376401 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.377050 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.378349 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.378813 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.378881 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.378893 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.379046 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.379076 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:06:06.379172 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.376055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:41975 (local address 127.14.33.3:39721)
0925 05:06:06.376113 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.376114 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.376123 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:06.376148 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.376149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.376158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.376208 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:06.376421 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.377163 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.378352 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.378882 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.378921 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.378929 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.379091 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.379123 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:06:06.379372 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.375861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:39721 (local address 127.14.33.1:41975)
0925 05:06:06.375958 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.375967 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.376003 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.376262 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.376263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.376264 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.376344 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.376346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.377272 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.377272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.378260 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.378263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.378973 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.378974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.379026 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.379029 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.379267 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.379310 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":21,"thread_start_us":62,"threads_started":1}
I20260925 05:06:06.379572 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624960bf89464d74a5284f6eb1d35d16" candidate_uuid: "15d71d75eb7c490daefff6b693cb181c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35bef191b3d4508bd71eb00c579f08c" is_pre_election: true
I20260925 05:06:06.379659 22402 raft_consensus.cc:2454] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d71d75eb7c490daefff6b693cb181c in term 0.
I20260925 05:06:06.379729 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.375691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44649 (local address 127.14.33.1:59061)
0925 05:06:06.375807 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.375818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.375866 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.376223 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.376224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.376225 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:06.376344 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.376347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.377141 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.377142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.378261 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.378263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.378916 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.378918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.378990 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.378993 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.379042 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.379086 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":30,"thread_start_us":77,"threads_started":1}
I20260925 05:06:06.379851 22056 leader_election.cc:302] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.379983 22484 raft_consensus.cc:2781] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:06.380023 22484 raft_consensus.cc:482] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:06.380053 22484 raft_consensus.cc:3037] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.380133 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624960bf89464d74a5284f6eb1d35d16" candidate_uuid: "15d71d75eb7c490daefff6b693cb181c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0" is_pre_election: true
I20260925 05:06:06.380224 22265 raft_consensus.cc:2454] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d71d75eb7c490daefff6b693cb181c in term 0.
I20260925 05:06:06.380903 22484 raft_consensus.cc:504] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.381002 22484 leader_election.cc:288] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [CANDIDATE]: Term 1 election: Requested vote from peers 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649), e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.381143 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624960bf89464d74a5284f6eb1d35d16" candidate_uuid: "15d71d75eb7c490daefff6b693cb181c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0"
I20260925 05:06:06.381214 22265 raft_consensus.cc:3037] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.381191 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624960bf89464d74a5284f6eb1d35d16" candidate_uuid: "15d71d75eb7c490daefff6b693cb181c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35bef191b3d4508bd71eb00c579f08c"
I20260925 05:06:06.381245 22402 raft_consensus.cc:3037] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.381910 22402 raft_consensus.cc:2454] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d71d75eb7c490daefff6b693cb181c in term 1.
I20260925 05:06:06.381973 22265 raft_consensus.cc:2454] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d71d75eb7c490daefff6b693cb181c in term 1.
I20260925 05:06:06.382057 22056 leader_election.cc:302] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.382131 22484 raft_consensus.cc:2781] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:06.382246 22484 raft_consensus.cc:686] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 1 LEADER]: Becoming Leader. State: Replica: 15d71d75eb7c490daefff6b693cb181c, State: Running, Role: LEADER
I20260925 05:06:06.382328 22484 consensus_queue.cc:237] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.382978 21968 catalog_manager.cc:5495] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c reported cstate change: term changed from 0 to 1, leader changed from <none> to 15d71d75eb7c490daefff6b693cb181c (127.14.33.1). New cstate: current_term: 1 leader_uuid: "15d71d75eb7c490daefff6b693cb181c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } health_report { overall_health: UNKNOWN } } }
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312765, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.62@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:06.417044 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.409766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:39063 (local address 127.0.0.1:47942)
0925 05:06:06.409953 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.409963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.410021 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.410159 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.410159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.410438 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:06.410485 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.410486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.411337 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.411338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.411893 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.411894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.412360 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.412360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.412401 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.412949 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:06.412954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:06.414292 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:06.415983 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:06.415990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:06.415991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:06.416173 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:06.416437 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:06.416438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:06.416438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:06.416499 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:06.416805 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:06.416810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:06.416848 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.416899 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.416976 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260925 05:06:06.417194 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.409856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.14.33.62:39063)
0925 05:06:06.409917 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.409919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.410021 (+ 102us) server_negotiation.cc:407] Connection header received
0925 05:06:06.410068 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.410069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.410079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.410146 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.410556 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.411259 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.411980 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.412277 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.412342 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.414388 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:06.414390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:06.414391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:06.414400 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:06.415903 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.416236 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.416237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.416238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.416294 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.416586 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.416587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.416587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.416741 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:06.416809 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.416952 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.417044 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:06:06.419145 21968 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:47942:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 05:06:06.419384 21968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:06.420614 22245 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6c18c745744d4bdf2c7b43ca81b5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e983c9509b04beca619b351f747b892]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:06.420683 22105 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6c18c745744d4bdf2c7b43ca81b5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e983c9509b04beca619b351f747b892]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:06.420773 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6c18c745744d4bdf2c7b43ca81b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.420799 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6c18c745744d4bdf2c7b43ca81b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.421253 22385 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6c18c745744d4bdf2c7b43ca81b5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e983c9509b04beca619b351f747b892]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 05:06:06.421377 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6c18c745744d4bdf2c7b43ca81b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.422155 22479 tablet_bootstrap.cc:492] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: Bootstrap starting.
I20260925 05:06:06.422155 22478 tablet_bootstrap.cc:492] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: Bootstrap starting.
I20260925 05:06:06.422464 22478 tablet_bootstrap.cc:654] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.422554 22479 tablet_bootstrap.cc:654] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.422672 22476 tablet_bootstrap.cc:492] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: Bootstrap starting.
I20260925 05:06:06.423038 22478 tablet_bootstrap.cc:492] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: No bootstrap required, opened a new log
I20260925 05:06:06.423066 22476 tablet_bootstrap.cc:654] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.423085 22478 ts_tablet_manager.cc:1397] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:06.423204 22478 raft_consensus.cc:348] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.423269 22478 raft_consensus.cc:374] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.423290 22478 raft_consensus.cc:729] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.423290 22479 tablet_bootstrap.cc:492] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: No bootstrap required, opened a new log
I20260925 05:06:06.423328 22479 ts_tablet_manager.cc:1397] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:06.423334 22478 consensus_queue.cc:260] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.423429 22478 ts_tablet_manager.cc:1428] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.423463 22479 raft_consensus.cc:348] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.423529 22479 raft_consensus.cc:374] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.423502 22488 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.423548 22479 raft_consensus.cc:729] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d71d75eb7c490daefff6b693cb181c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.423564 22488 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.423589 22479 consensus_queue.cc:260] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.423669 22479 ts_tablet_manager.cc:1428] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.423784 22484 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.423878 22484 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.424258 22476 tablet_bootstrap.cc:492] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: No bootstrap required, opened a new log
I20260925 05:06:06.424302 22476 ts_tablet_manager.cc:1397] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:06:06.424592 22476 raft_consensus.cc:348] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.424656 22476 raft_consensus.cc:374] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.424672 22476 raft_consensus.cc:729] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.424712 22476 consensus_queue.cc:260] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.424779 22476 ts_tablet_manager.cc:1428] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.424825 22481 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.424938 22481 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.572633 22488 raft_consensus.cc:482] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:06.572708 22488 raft_consensus.cc:504] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.572883 22488 leader_election.cc:288] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.573081 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6c18c745744d4bdf2c7b43ca81b5d" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c" is_pre_election: true
I20260925 05:06:06.573081 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6c18c745744d4bdf2c7b43ca81b5d" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35bef191b3d4508bd71eb00c579f08c" is_pre_election: true
I20260925 05:06:06.573200 22402 raft_consensus.cc:2454] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 0.
I20260925 05:06:06.573200 22125 raft_consensus.cc:2454] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 0.
I20260925 05:06:06.573346 22196 leader_election.cc:302] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d67995f08d140ceaa09a80f469903e0, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.573458 22488 raft_consensus.cc:2781] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:06.573496 22488 raft_consensus.cc:482] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:06.573522 22488 raft_consensus.cc:3037] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.574208 22488 raft_consensus.cc:504] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.574343 22488 leader_election.cc:288] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 election: Requested vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721)
I20260925 05:06:06.574472 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6c18c745744d4bdf2c7b43ca81b5d" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c"
I20260925 05:06:06.574517 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6c18c745744d4bdf2c7b43ca81b5d" candidate_uuid: "6d67995f08d140ceaa09a80f469903e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35bef191b3d4508bd71eb00c579f08c"
I20260925 05:06:06.574551 22125 raft_consensus.cc:3037] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.574576 22402 raft_consensus.cc:3037] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.575258 22402 raft_consensus.cc:2454] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 1.
I20260925 05:06:06.575265 22125 raft_consensus.cc:2454] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d67995f08d140ceaa09a80f469903e0 in term 1.
I20260925 05:06:06.575426 22199 leader_election.cc:302] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, 6d67995f08d140ceaa09a80f469903e0; no voters:
I20260925 05:06:06.575558 22488 raft_consensus.cc:2781] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:06.575613 22488 raft_consensus.cc:686] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Running, Role: LEADER
I20260925 05:06:06.575665 22488 consensus_queue.cc:237] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.575963 22483 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: TxnStatusTablet state changed. Reason: New leader 6d67995f08d140ceaa09a80f469903e0. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.576038 22483 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:06.576164 22520 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 05:06:06.576292 22520 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 05:06:06.576251 21967 catalog_manager.cc:5495] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2). New cstate: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } health_report { overall_health: UNKNOWN } } }
I20260925 05:06:06.607867 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.605327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44649 (local address 127.0.0.1:46814)
0925 05:06:06.605388 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.605399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.605446 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.605691 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.605692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.605693 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:06.605734 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.605736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.606529 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.606530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.607054 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.607056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.607515 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.607516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.607558 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.607571 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.607736 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.607742 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.607781 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.607815 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 05:06:06.607878 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.605516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.14.33.2:44649)
0925 05:06:06.605576 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.605578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.605586 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:06.605608 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.605609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.605620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.605667 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:06.605789 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.606467 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.607134 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.607447 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.607517 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.607648 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.607680 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.607732 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.607785 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.607806 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:06:06.608786 22402 raft_consensus.cc:1261] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Refusing update from remote peer 6d67995f08d140ceaa09a80f469903e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.608853 22125 raft_consensus.cc:1261] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Refusing update from remote peer 6d67995f08d140ceaa09a80f469903e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.609000 22483 consensus_queue.cc:1035] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.609220 22483 consensus_queue.cc:1035] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.609573 22481 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: TxnStatusTablet state changed. Reason: New leader 6d67995f08d140ceaa09a80f469903e0. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.609658 22481 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.609664 22484 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: TxnStatusTablet state changed. Reason: New leader 6d67995f08d140ceaa09a80f469903e0. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.609755 22484 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.610169 22483 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.610240 22483 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:06.610322 22491 mvcc.cc:205] Tried to move back new op lower bound from 7333121092028186624 to 7333121091894472704. Current Snapshot: MvccSnapshot[applied={T|T < 7333121092028186624}]
I20260925 05:06:06.610424 22488 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.610499 22488 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 6d67995f08d140ceaa09a80f469903e0: This TxnStatusTablet replica's current role is: LEADER
I20260925 05:06:06.610930 22484 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.611017 22484 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P 15d71d75eb7c490daefff6b693cb181c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.611286 22481 tablet_replica.cc:439] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d67995f08d140ceaa09a80f469903e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } } }
I20260925 05:06:06.611366 22481 tablet_replica.cc:442] T 0bf6c18c745744d4bdf2c7b43ca81b5d P e35bef191b3d4508bd71eb00c579f08c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 05:06:06.614590 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.612164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.14.33.1:45663)
0925 05:06:06.612195 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.612197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.612204 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:06.612268 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.612269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.612283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.612353 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:06.612469 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.612893 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.613910 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.614088 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.614152 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.614299 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.614392 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.614458 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.614526 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.614543 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:06:06.620036 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.617006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:60565 (local address 127.14.33.2:44649)
0925 05:06:06.617062 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.617064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.617182 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:06:06.617238 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.617239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.617247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.617291 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:06.617416 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.617821 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.619434 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.619604 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.619665 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.619840 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.619864 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.619901 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.619974 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.619991 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:06:06.620101 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.616951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44649 (local address 127.14.33.1:60565)
0925 05:06:06.617101 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.617111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.617188 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.617304 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.617305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.617306 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:06.617364 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.617365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.617919 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.617920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.619349 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.619352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.619681 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.619682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.619744 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.619759 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.619918 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.619920 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.619972 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.620013 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260925 05:06:06.622629 22265 raft_consensus.cc:1261] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Refusing update from remote peer 15d71d75eb7c490daefff6b693cb181c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.622660 22402 raft_consensus.cc:1261] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Refusing update from remote peer 15d71d75eb7c490daefff6b693cb181c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.622902 22508 consensus_queue.cc:1035] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.623075 22484 consensus_queue.cc:1035] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.631033 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.628698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:38489 (local address 127.14.33.1:45663)
0925 05:06:06.628736 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.628738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.628875 (+ 137us) server_negotiation.cc:407] Connection header received
0925 05:06:06.628923 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.628924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.628931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.628974 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:06.629065 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.629506 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.630443 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.630598 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.630646 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.630806 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.630828 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.630912 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.630976 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.630991 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:06:06.631071 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.628656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45663 (local address 127.14.33.2:38489)
0925 05:06:06.628807 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:06.628818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:06.628873 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:06.628979 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:06.628979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:06.628980 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:06.629015 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.629016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.629571 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.629572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.630375 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:06.630377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.630656 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.630657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:06.630725 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.630737 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.630920 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.630922 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:06:06.630972 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:06:06.631008 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260925 05:06:06.637058 21968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:06.637164 21968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:06.637764 21968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Sending DeleteTablet for 3 replicas of tablet 624960bf89464d74a5284f6eb1d35d16
I20260925 05:06:06.638003 22245 tablet_service.cc:1514] Processing DeleteTablet for tablet 624960bf89464d74a5284f6eb1d35d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46796
I20260925 05:06:06.638054 22385 tablet_service.cc:1514] Processing DeleteTablet for tablet 624960bf89464d74a5284f6eb1d35d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:48612
I20260925 05:06:06.638088 22501 tablet_replica.cc:330] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: stopping tablet replica
I20260925 05:06:06.638136 22501 raft_consensus.cc:2229] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:06.638149 22502 tablet_replica.cc:330] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: stopping tablet replica
I20260925 05:06:06.638172 22501 raft_consensus.cc:2258] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.638195 22502 raft_consensus.cc:2229] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:06.638232 22502 raft_consensus.cc:2258] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.638290 22105 tablet_service.cc:1514] Processing DeleteTablet for tablet 624960bf89464d74a5284f6eb1d35d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:47402
I20260925 05:06:06.638401 22500 tablet_replica.cc:330] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: stopping tablet replica
I20260925 05:06:06.638447 22500 raft_consensus.cc:2229] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:06.638517 22500 raft_consensus.cc:2258] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.638751 22501 ts_tablet_manager.cc:1905] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.638756 22500 ts_tablet_manager.cc:1905] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.640301 22500 ts_tablet_manager.cc:1918] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:06.640347 22500 log.cc:1196] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/624960bf89464d74a5284f6eb1d35d16
I20260925 05:06:06.640493 22501 ts_tablet_manager.cc:1918] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:06.640542 22501 log.cc:1196] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/624960bf89464d74a5284f6eb1d35d16
I20260925 05:06:06.640568 22500 ts_tablet_manager.cc:1939] T 624960bf89464d74a5284f6eb1d35d16 P 15d71d75eb7c490daefff6b693cb181c: Deleting consensus metadata
I20260925 05:06:06.640758 22501 ts_tablet_manager.cc:1939] T 624960bf89464d74a5284f6eb1d35d16 P 6d67995f08d140ceaa09a80f469903e0: Deleting consensus metadata
I20260925 05:06:06.641010 21952 catalog_manager.cc:4841] TS 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649): tablet 624960bf89464d74a5284f6eb1d35d16 (table test-table [id=842ca062cab047abad9473e069d12768]) successfully deleted
I20260925 05:06:06.641093 21955 catalog_manager.cc:4841] TS 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663): tablet 624960bf89464d74a5284f6eb1d35d16 (table test-table [id=842ca062cab047abad9473e069d12768]) successfully deleted
I20260925 05:06:06.641968 22502 ts_tablet_manager.cc:1905] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.643378 22502 ts_tablet_manager.cc:1918] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:06.643422 22502 log.cc:1196] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/624960bf89464d74a5284f6eb1d35d16
I20260925 05:06:06.643613 22502 ts_tablet_manager.cc:1939] T 624960bf89464d74a5284f6eb1d35d16 P e35bef191b3d4508bd71eb00c579f08c: Deleting consensus metadata
I20260925 05:06:06.643913 21952 catalog_manager.cc:4841] TS e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721): tablet 624960bf89464d74a5284f6eb1d35d16 (table test-table [id=842ca062cab047abad9473e069d12768]) successfully deleted
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:06:06.676471 22017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:06.670057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.14.33.62:39063)
0925 05:06:06.670098 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.670100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.670227 (+ 127us) server_negotiation.cc:407] Connection header received
0925 05:06:06.670284 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.670285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.670297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.670353 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.671107 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.671852 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.672886 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.673179 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.673241 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.676410 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47948: BlockingRecv error: recv got EOF from 127.0.0.1:47948 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 05:06:06 dist-test-slave-4dmq krb5kdc[21923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM
I20260925 05:06:06.685228 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.677165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.14.33.62:39063)
0925 05:06:06.677202 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.677203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.677418 (+ 215us) server_negotiation.cc:407] Connection header received
0925 05:06:06.677464 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.677465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.677474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.677522 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:06.678013 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.678685 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.679588 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.679883 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.679950 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.682338 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:06.682339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:06.682340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:06.682346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:06.683931 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.684323 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.684324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.684324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.684385 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:06.684590 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:06.684591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:06.684592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:06.684719 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:06.684861 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.684997 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.685071 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:06:06.687023 21968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:06.687191 21968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:06.688535 22245 tablet_service.cc:1467] Processing CreateTablet for tablet d2996c85b2b94803afcfc028fc75481d (DEFAULT_TABLE table=test-table [id=cb2808daf5a145cc8943ee33bbba7c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.688660 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2996c85b2b94803afcfc028fc75481d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.688622 22105 tablet_service.cc:1467] Processing CreateTablet for tablet d2996c85b2b94803afcfc028fc75481d (DEFAULT_TABLE table=test-table [id=cb2808daf5a145cc8943ee33bbba7c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.688755 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2996c85b2b94803afcfc028fc75481d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.689206 22385 tablet_service.cc:1467] Processing CreateTablet for tablet d2996c85b2b94803afcfc028fc75481d (DEFAULT_TABLE table=test-table [id=cb2808daf5a145cc8943ee33bbba7c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.689316 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2996c85b2b94803afcfc028fc75481d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.690002 22478 tablet_bootstrap.cc:492] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Bootstrap starting.
I20260925 05:06:06.690292 22478 tablet_bootstrap.cc:654] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.690539 22476 tablet_bootstrap.cc:492] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Bootstrap starting.
I20260925 05:06:06.690905 22476 tablet_bootstrap.cc:654] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.692365 22479 tablet_bootstrap.cc:492] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Bootstrap starting.
I20260925 05:06:06.692595 22476 tablet_bootstrap.cc:492] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: No bootstrap required, opened a new log
I20260925 05:06:06.692636 22476 ts_tablet_manager.cc:1397] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:06:06.692806 22479 tablet_bootstrap.cc:654] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.692823 22476 raft_consensus.cc:348] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.692891 22476 raft_consensus.cc:374] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.692910 22476 raft_consensus.cc:729] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.692952 22476 consensus_queue.cc:260] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.693042 22476 ts_tablet_manager.cc:1428] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:06.693583 22479 tablet_bootstrap.cc:492] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: No bootstrap required, opened a new log
I20260925 05:06:06.693630 22479 ts_tablet_manager.cc:1397] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:06.693754 22479 raft_consensus.cc:348] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.693813 22479 raft_consensus.cc:374] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.693832 22479 raft_consensus.cc:729] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d71d75eb7c490daefff6b693cb181c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.693873 22479 consensus_queue.cc:260] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.693949 22479 ts_tablet_manager.cc:1428] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.694016 22478 tablet_bootstrap.cc:492] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: No bootstrap required, opened a new log
I20260925 05:06:06.694089 22478 ts_tablet_manager.cc:1397] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 05:06:06.694193 22478 raft_consensus.cc:348] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.694263 22478 raft_consensus.cc:374] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.694978 22478 raft_consensus.cc:729] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.695026 22478 consensus_queue.cc:260] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.695189 22478 ts_tablet_manager.cc:1428] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:06:06.916100 22481 raft_consensus.cc:482] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:06.916203 22481 raft_consensus.cc:504] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.916391 22481 leader_election.cc:288] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649)
I20260925 05:06:06.916625 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2996c85b2b94803afcfc028fc75481d" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c" is_pre_election: true
I20260925 05:06:06.916617 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2996c85b2b94803afcfc028fc75481d" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0" is_pre_election: true
I20260925 05:06:06.916723 22265 raft_consensus.cc:2454] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 0.
I20260925 05:06:06.916723 22125 raft_consensus.cc:2454] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 0.
I20260925 05:06:06.916870 22336 leader_election.cc:302] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d67995f08d140ceaa09a80f469903e0, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.916993 22481 raft_consensus.cc:2781] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:06.917057 22481 raft_consensus.cc:482] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:06.917081 22481 raft_consensus.cc:3037] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.917778 22481 raft_consensus.cc:504] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.917917 22481 leader_election.cc:288] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 election: Requested vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649)
I20260925 05:06:06.918082 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2996c85b2b94803afcfc028fc75481d" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0"
I20260925 05:06:06.918092 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2996c85b2b94803afcfc028fc75481d" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c"
I20260925 05:06:06.918171 22265 raft_consensus.cc:3037] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.918198 22125 raft_consensus.cc:3037] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.918776 22125 raft_consensus.cc:2454] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 1.
I20260925 05:06:06.918782 22265 raft_consensus.cc:2454] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 1.
I20260925 05:06:06.918967 22336 leader_election.cc:302] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d67995f08d140ceaa09a80f469903e0, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.919065 22481 raft_consensus.cc:2781] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:06.919170 22481 raft_consensus.cc:686] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 1 LEADER]: Becoming Leader. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Running, Role: LEADER
I20260925 05:06:06.919231 22481 consensus_queue.cc:237] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.919849 21968 catalog_manager.cc:5495] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c reported cstate change: term changed from 0 to 1, leader changed from <none> to e35bef191b3d4508bd71eb00c579f08c (127.14.33.3). New cstate: current_term: 1 leader_uuid: "e35bef191b3d4508bd71eb00c579f08c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:06.976786 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:06.974568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.14.33.3:39721)
0925 05:06:06.974686 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:06.974688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:06.974697 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:06.974738 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:06.974738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:06.974756 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:06.974826 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:06.974949 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.975413 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.976104 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:06.976294 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:06.976380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:06.976504 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:06.976604 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:06.976658 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:06.976696 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 05:06:06.976712 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260925 05:06:06.977674 22265 raft_consensus.cc:1261] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Refusing update from remote peer e35bef191b3d4508bd71eb00c579f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.977746 22125 raft_consensus.cc:1261] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Refusing update from remote peer e35bef191b3d4508bd71eb00c579f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:06.977953 22551 consensus_queue.cc:1035] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.978088 22481 consensus_queue.cc:1035] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:06.981076 21968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:06.981174 21968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:06.981719 21968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Sending DeleteTablet for 3 replicas of tablet d2996c85b2b94803afcfc028fc75481d
I20260925 05:06:06.981941 22105 tablet_service.cc:1514] Processing DeleteTablet for tablet d2996c85b2b94803afcfc028fc75481d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:47402
I20260925 05:06:06.981982 22385 tablet_service.cc:1514] Processing DeleteTablet for tablet d2996c85b2b94803afcfc028fc75481d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:48612
I20260925 05:06:06.981987 22245 tablet_service.cc:1514] Processing DeleteTablet for tablet d2996c85b2b94803afcfc028fc75481d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:06 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46796
I20260925 05:06:06.982077 22500 tablet_replica.cc:330] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: stopping tablet replica
I20260925 05:06:06.982132 22500 raft_consensus.cc:2229] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:06.982168 22500 raft_consensus.cc:2258] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.982237 22502 tablet_replica.cc:330] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: stopping tablet replica
I20260925 05:06:06.982288 22502 raft_consensus.cc:2229] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:06.982364 22502 raft_consensus.cc:2258] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.982434 22501 tablet_replica.cc:330] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: stopping tablet replica
I20260925 05:06:06.982486 22501 raft_consensus.cc:2229] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:06.982447 21968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 05:06:06.982524 22501 raft_consensus.cc:2258] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:06.982546 22500 ts_tablet_manager.cc:1905] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 05:06:06.982568 21968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:06.983269 22501 ts_tablet_manager.cc:1905] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.984047 22500 ts_tablet_manager.cc:1918] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:06.984107 22500 log.cc:1196] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/d2996c85b2b94803afcfc028fc75481d
I20260925 05:06:06.984335 22500 ts_tablet_manager.cc:1939] T d2996c85b2b94803afcfc028fc75481d P 15d71d75eb7c490daefff6b693cb181c: Deleting consensus metadata
I20260925 05:06:06.984426 22501 ts_tablet_manager.cc:1918] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:06.984483 22501 log.cc:1196] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/d2996c85b2b94803afcfc028fc75481d
I20260925 05:06:06.984638 21955 catalog_manager.cc:4841] TS 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663): tablet d2996c85b2b94803afcfc028fc75481d (table test-table [id=cb2808daf5a145cc8943ee33bbba7c97]) successfully deleted
I20260925 05:06:06.984689 22501 ts_tablet_manager.cc:1939] T d2996c85b2b94803afcfc028fc75481d P 6d67995f08d140ceaa09a80f469903e0: Deleting consensus metadata
I20260925 05:06:06.985018 21952 catalog_manager.cc:4841] TS 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649): tablet d2996c85b2b94803afcfc028fc75481d (table test-table [id=cb2808daf5a145cc8943ee33bbba7c97]) successfully deleted
I20260925 05:06:06.985044 22502 ts_tablet_manager.cc:1905] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:06.986130 22105 tablet_service.cc:1467] Processing CreateTablet for tablet 1860f2c20878434d9af75dfaf0ca75d6 (DEFAULT_TABLE table=test-table [id=e7942e3e81d145e285a80a478a520fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.986140 22245 tablet_service.cc:1467] Processing CreateTablet for tablet 1860f2c20878434d9af75dfaf0ca75d6 (DEFAULT_TABLE table=test-table [id=e7942e3e81d145e285a80a478a520fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.986240 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1860f2c20878434d9af75dfaf0ca75d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.986263 22245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1860f2c20878434d9af75dfaf0ca75d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.986599 22502 ts_tablet_manager.cc:1918] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:06.986646 22502 log.cc:1196] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/d2996c85b2b94803afcfc028fc75481d
I20260925 05:06:06.986853 22502 ts_tablet_manager.cc:1939] T d2996c85b2b94803afcfc028fc75481d P e35bef191b3d4508bd71eb00c579f08c: Deleting consensus metadata
I20260925 05:06:06.987185 22385 tablet_service.cc:1467] Processing CreateTablet for tablet 1860f2c20878434d9af75dfaf0ca75d6 (DEFAULT_TABLE table=test-table [id=e7942e3e81d145e285a80a478a520fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:06.987265 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1860f2c20878434d9af75dfaf0ca75d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:06.987460 22478 tablet_bootstrap.cc:492] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Bootstrap starting.
I20260925 05:06:06.987520 22479 tablet_bootstrap.cc:492] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c: Bootstrap starting.
I20260925 05:06:06.987730 22478 tablet_bootstrap.cc:654] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.987782 22479 tablet_bootstrap.cc:654] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.988314 22479 tablet_bootstrap.cc:492] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c: No bootstrap required, opened a new log
I20260925 05:06:06.988353 22479 ts_tablet_manager.cc:1397] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:06:06.988425 22476 tablet_bootstrap.cc:492] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Bootstrap starting.
I20260925 05:06:06.988461 22479 raft_consensus.cc:348] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.988536 22479 raft_consensus.cc:374] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.988562 22479 raft_consensus.cc:729] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d71d75eb7c490daefff6b693cb181c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.988600 22479 consensus_queue.cc:260] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.988718 22479 ts_tablet_manager.cc:1428] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.988817 22476 tablet_bootstrap.cc:654] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:06.989069 22478 tablet_bootstrap.cc:492] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: No bootstrap required, opened a new log
I20260925 05:06:06.989118 22478 ts_tablet_manager.cc:1397] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:06.989262 21952 catalog_manager.cc:4841] TS e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721): tablet d2996c85b2b94803afcfc028fc75481d (table test-table [id=cb2808daf5a145cc8943ee33bbba7c97]) successfully deleted
I20260925 05:06:06.989241 22478 raft_consensus.cc:348] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.989318 22478 raft_consensus.cc:374] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.989339 22478 raft_consensus.cc:729] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d67995f08d140ceaa09a80f469903e0, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.989384 22478 consensus_queue.cc:260] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.989459 22476 tablet_bootstrap.cc:492] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: No bootstrap required, opened a new log
I20260925 05:06:06.989470 22478 ts_tablet_manager.cc:1428] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.989495 22476 ts_tablet_manager.cc:1397] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:06.989636 22476 raft_consensus.cc:348] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.989694 22476 raft_consensus.cc:374] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:06.989713 22476 raft_consensus.cc:729] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Initialized, Role: FOLLOWER
I20260925 05:06:06.989770 22476 consensus_queue.cc:260] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.989876 22476 ts_tablet_manager.cc:1428] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:06.991051 22557 raft_consensus.cc:482] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:06.991123 22557 raft_consensus.cc:504] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.991271 22557 leader_election.cc:288] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649)
I20260925 05:06:06.991380 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1860f2c20878434d9af75dfaf0ca75d6" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c" is_pre_election: true
I20260925 05:06:06.991400 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1860f2c20878434d9af75dfaf0ca75d6" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0" is_pre_election: true
I20260925 05:06:06.991448 22125 raft_consensus.cc:2454] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 0.
I20260925 05:06:06.991466 22265 raft_consensus.cc:2454] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 0.
I20260925 05:06:06.991572 22339 leader_election.cc:302] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.991657 22557 raft_consensus.cc:2781] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:06.991689 22557 raft_consensus.cc:482] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:06.991710 22557 raft_consensus.cc:3037] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.992429 22557 raft_consensus.cc:504] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.992599 22557 leader_election.cc:288] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 election: Requested vote from peers 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663), 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649)
I20260925 05:06:06.992640 22125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1860f2c20878434d9af75dfaf0ca75d6" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d71d75eb7c490daefff6b693cb181c"
I20260925 05:06:06.992704 22125 raft_consensus.cc:3037] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.992743 22265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1860f2c20878434d9af75dfaf0ca75d6" candidate_uuid: "e35bef191b3d4508bd71eb00c579f08c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d67995f08d140ceaa09a80f469903e0"
I20260925 05:06:06.992798 22265 raft_consensus.cc:3037] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:06.993356 22125 raft_consensus.cc:2454] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 1.
I20260925 05:06:06.993469 22265 raft_consensus.cc:2454] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35bef191b3d4508bd71eb00c579f08c in term 1.
I20260925 05:06:06.993517 22339 leader_election.cc:302] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d71d75eb7c490daefff6b693cb181c, e35bef191b3d4508bd71eb00c579f08c; no voters:
I20260925 05:06:06.993611 22557 raft_consensus.cc:2781] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:06.993678 22557 raft_consensus.cc:686] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 1 LEADER]: Becoming Leader. State: Replica: e35bef191b3d4508bd71eb00c579f08c, State: Running, Role: LEADER
I20260925 05:06:06.993721 22557 consensus_queue.cc:237] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } }
I20260925 05:06:06.994107 21967 catalog_manager.cc:5495] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c reported cstate change: term changed from 0 to 1, leader changed from <none> to e35bef191b3d4508bd71eb00c579f08c (127.14.33.3). New cstate: current_term: 1 leader_uuid: "e35bef191b3d4508bd71eb00c579f08c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35bef191b3d4508bd71eb00c579f08c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 39721 } health_report { overall_health: HEALTHY } } }
I20260925 05:06:07.003974 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.000932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44649 (local address 127.14.33.3:35941)
0925 05:06:07.001067 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:07.001078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:07.001212 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:07.001366 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:07.001367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:07.001368 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:07.001424 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:07.001425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.002262 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.002263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:07.003066 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:07.003087 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.003567 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.003568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:07.003627 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:07.003645 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:07.003836 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:07.003837 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:06:07.003874 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 05:06:07.003906 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 05:06:07.003973 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.000997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:35941 (local address 127.14.33.2:44649)
0925 05:06:07.001037 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:07.001040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:07.001157 (+ 117us) server_negotiation.cc:407] Connection header received
0925 05:06:07.001288 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:07.001289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:07.001300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:07.001362 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:07.001472 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.002185 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.003156 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.003485 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.003555 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:07.003744 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:07.003778 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:07.003830 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:07.003889 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 05:06:07.003908 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:06:07.005674 22265 raft_consensus.cc:1261] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Refusing update from remote peer e35bef191b3d4508bd71eb00c579f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:07.005699 22125 raft_consensus.cc:1261] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Refusing update from remote peer e35bef191b3d4508bd71eb00c579f08c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:07.005877 22551 consensus_queue.cc:1035] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d67995f08d140ceaa09a80f469903e0" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:07.005956 22557 consensus_queue.cc:1035] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d71d75eb7c490daefff6b693cb181c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:07.012954 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.010588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:51727 (local address 127.14.33.3:39721)
0925 05:06:07.010630 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:07.010632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:07.010639 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:07.010659 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:07.010660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:07.010670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:07.010720 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:07.010820 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.011237 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.012221 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.012454 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.012505 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:07.012690 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:07.012722 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:07.012767 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:07.012876 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 05:06:07.012894 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:06:07.012956 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.010414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:39721 (local address 127.14.33.2:51727)
0925 05:06:07.010492 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:07.010502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:07.010535 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:07.010726 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:07.010727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:07.010728 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 05:06:07.010767 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:07.010768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.011301 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.011302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:07.012141 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:07.012143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:07.012521 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:07.012522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:07.012578 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:07.012584 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 05:06:07.012792 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 05:06:07.012794 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:06:07.012840 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:06:07.012890 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 05:06:07.019055 21967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:07.019140 21967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:07.019716 21967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee4c126c743947648d98e741f8ad143c: Sending DeleteTablet for 3 replicas of tablet 1860f2c20878434d9af75dfaf0ca75d6
I20260925 05:06:07.019990 22385 tablet_service.cc:1514] Processing DeleteTablet for tablet 1860f2c20878434d9af75dfaf0ca75d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:07 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:48612
I20260925 05:06:07.020000 22105 tablet_service.cc:1514] Processing DeleteTablet for tablet 1860f2c20878434d9af75dfaf0ca75d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:07 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:47402
I20260925 05:06:07.020073 22500 tablet_replica.cc:330] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c: stopping tablet replica
I20260925 05:06:07.020102 22502 tablet_replica.cc:330] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: stopping tablet replica
I20260925 05:06:07.020117 22500 raft_consensus.cc:2229] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:07.020145 22500 raft_consensus.cc:2258] T 1860f2c20878434d9af75dfaf0ca75d6 P 15d71d75eb7c490daefff6b693cb181c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:07.020148 22502 raft_consensus.cc:2229] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:06:07.020200 22502 raft_consensus.cc:2258] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:07.020252 22245 tablet_service.cc:1514] Processing DeleteTablet for tablet 1860f2c20878434d9af75dfaf0ca75d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:07 UTC) from {username='oryx', principal='oryx/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46796
I20260925 05:06:07.020321 22501 tablet_replica.cc:330] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: stopping tablet replica
I20260925 05:06:07.020366 22501 raft_consensus.cc:2229] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:07.020397 22501 raft_consensus.cc:2258] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:07.020423 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22040
I20260925 05:06:07.021548 22501 ts_tablet_manager.cc:1905] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:07.022699 22502 ts_tablet_manager.cc:1905] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:07.022992 22501 ts_tablet_manager.cc:1918] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:07.023049 22501 log.cc:1196] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/1860f2c20878434d9af75dfaf0ca75d6
I20260925 05:06:07.023262 22501 ts_tablet_manager.cc:1939] T 1860f2c20878434d9af75dfaf0ca75d6 P 6d67995f08d140ceaa09a80f469903e0: Deleting consensus metadata
I20260925 05:06:07.023550 21952 catalog_manager.cc:4841] TS 6d67995f08d140ceaa09a80f469903e0 (127.14.33.2:44649): tablet 1860f2c20878434d9af75dfaf0ca75d6 (table test-table [id=e7942e3e81d145e285a80a478a520fa1]) successfully deleted
I20260925 05:06:07.023969 22502 ts_tablet_manager.cc:1918] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 05:06:07.024034 22502 log.cc:1196] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/1860f2c20878434d9af75dfaf0ca75d6
I20260925 05:06:07.024211 22502 ts_tablet_manager.cc:1939] T 1860f2c20878434d9af75dfaf0ca75d6 P e35bef191b3d4508bd71eb00c579f08c: Deleting consensus metadata
I20260925 05:06:07.024501 21952 catalog_manager.cc:4841] TS e35bef191b3d4508bd71eb00c579f08c (127.14.33.3:39721): tablet 1860f2c20878434d9af75dfaf0ca75d6 (table test-table [id=e7942e3e81d145e285a80a478a520fa1]) successfully deleted
W20260925 05:06:07.028362 21955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:45663 (error 108)
I20260925 05:06:07.028518 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22180
I20260925 05:06:07.030408 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:07.030213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:45663 (local address 127.0.0.1:47420)
0925 05:06:07.030336 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:07.030379 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260925 05:06:07.033211 21955 catalog_manager.cc:4584] TS 15d71d75eb7c490daefff6b693cb181c (127.14.33.1:45663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1860f2c20878434d9af75dfaf0ca75d6: Network error: Client connection negotiation failed: client connection to 127.14.33.1:45663: connect: Connection refused (error 111)
I20260925 05:06:07.035748 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22320
I20260925 05:06:07.039697 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 21936
2026-09-25T05:06:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2636 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:07 dist-test-slave-4dmq krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:07 dist-test-slave-4dmq krb5kdc[22572](info): set up 2 sockets
Sep 25 05:06:07 dist-test-slave-4dmq krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260925 05:06:08.018558 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:06:08 dist-test-slave-4dmq krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:06:08.123456 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42933
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:08.207063 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:08.207307 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:08.207330 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:08.208909 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:08.208957 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:08.208978 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:08.209005 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:08.209029 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:08.210393 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:42933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:08.210640 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:08.210855 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:08.215123 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:08.215161 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:08.215350 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:08.215328 22585 server_base.cc:1034] running on GCE node
I20260925 05:06:08.215557 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:08.215773 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:08.216902 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790312768216891 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T05:06:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1180 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:08 dist-test-slave-4dmq krb5kdc[22598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:08 dist-test-slave-4dmq krb5kdc[22598](info): set up 2 sockets
Sep 25 05:06:08 dist-test-slave-4dmq krb5kdc[22598](info): commencing operation
krb5kdc: starting...
W20260925 05:06:09.206789 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:06:09.311671 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42263
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:09.386504 22611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:09.386694 22611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:09.386715 22611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:09.387809 22611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:09.387858 22611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:09.387871 22611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:09.387909 22611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:09.387926 22611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:09.389377 22611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:09.389595 22611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:09.389796 22611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:09.394357 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.394457 22619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.394414 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:09.394641 22611 server_base.cc:1034] running on GCE node
I20260925 05:06:09.394883 22611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:09.395136 22611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:09.396279 22611 hybrid_clock.cc:648] HybridClock initialized: now 1790312769396270 us; error 25 us; skew 500 ppm
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:09.398454 22611 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:06:09.398844 22611 webserver.cc:460] Webserver started at http://127.14.33.62:39979/ using document root <none> and password file <none>
I20260925 05:06:09.398996 22611 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:09.399030 22611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:09.399103 22611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:09.399931 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "95b604d5c3c64cc191edf0f0ff9b9f04"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "89ee67d39f4b9e35c87993cc602b3ee0"
server_key_iv: "3b2e53d277b2af2f6071717c540f3e0e"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.400188 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "95b604d5c3c64cc191edf0f0ff9b9f04"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "89ee67d39f4b9e35c87993cc602b3ee0"
server_key_iv: "3b2e53d277b2af2f6071717c540f3e0e"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.401346 22611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:06:09.401988 22626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.402160 22611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:09.402235 22611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "95b604d5c3c64cc191edf0f0ff9b9f04"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "89ee67d39f4b9e35c87993cc602b3ee0"
server_key_iv: "3b2e53d277b2af2f6071717c540f3e0e"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.402287 22611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:09.414291 22611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:09.416352 22611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:09.416455 22611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:09.420130 22611 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:42263
I20260925 05:06:09.420156 22678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:42263 every 8 connection(s)
I20260925 05:06:09.420459 22611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:09.421260 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:09.423178 22679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Bootstrap starting.
I20260925 05:06:09.423691 22679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:09.423997 22679 log.cc:824] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:09.424636 22679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: No bootstrap required, opened a new log
I20260925 05:06:09.425313 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22611
I20260925 05:06:09.425417 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:06:09.425582 14468 external_mini_cluster.cc:1428] Setting key a3c44df9b561b41fe253b9e64a0114ca
I20260925 05:06:09.426589 22679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:09.426736 22679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:09.426852 22679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b604d5c3c64cc191edf0f0ff9b9f04, State: Initialized, Role: FOLLOWER
I20260925 05:06:09.426992 22679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:09.427066 22679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:09.427088 22679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:09.427145 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:09.428139 22679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:09.428236 22679 leader_election.cc:302] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b604d5c3c64cc191edf0f0ff9b9f04; no voters:
I20260925 05:06:09.428369 22679 leader_election.cc:288] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:09.428575 22679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:09.428550 22684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:09.428750 22684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 LEADER]: Becoming Leader. State: Replica: 95b604d5c3c64cc191edf0f0ff9b9f04, State: Running, Role: LEADER
I20260925 05:06:09.428879 22684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:09.430135 22686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } } }
I20260925 05:06:09.430294 22686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: This master's current role is: LEADER
W20260925 05:06:09.430410 22698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:09.430490 22698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:06:09.430850 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b604d5c3c64cc191edf0f0ff9b9f04. Latest consensus state: current_term: 1 leader_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } } }
I20260925 05:06:09.430905 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:09.430943 22701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:09.431102 22701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:09.432516 22701 catalog_manager.cc:1347] Generated new cluster ID: bf0bd01a4bbc46d5a49420886d36ba66
I20260925 05:06:09.432566 22701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:09.436786 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.426013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.14.33.62:42263)
0925 05:06:09.426190 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:09.426199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:09.426228 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:06:09.426552 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:09.426559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:09.426614 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:09.426740 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:09.427551 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.428915 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.429818 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.430430 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.430533 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.433254 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:09.433258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:09.433258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:09.433269 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:09.435084 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.435488 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.435491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.435492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.435616 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.435816 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.435817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.435817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.436117 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:09.436264 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:09.436360 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 05:06:09.436580 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260925 05:06:09.442426 22701 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:09.443058 22701 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:09.450974 22701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Generated new TSK 0
I20260925 05:06:09.451115 22701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:06:09.479439 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:42263
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:09.554631 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:09.554818 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:09.554852 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:09.555827 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:09.555887 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:09.555927 22705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:09.557190 22705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:09.557416 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:09.557608 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:09.562305 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.562301 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:09.562333 22705 server_base.cc:1034] running on GCE node
W20260925 05:06:09.562309 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:09.562749 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:09.562976 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:09.564138 22705 hybrid_clock.cc:648] HybridClock initialized: now 1790312769564129 us; error 37 us; skew 500 ppm
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:09.571348 22705 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:06:09.571730 22705 webserver.cc:460] Webserver started at http://127.14.33.1:39593/ using document root <none> and password file <none>
I20260925 05:06:09.571902 22705 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:09.571935 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:09.572022 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:09.572837 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "d77a2d1431e44e179010a04cb7a59b7f"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "c44c4595b085e6039f80df0525c41ceb"
server_key_iv: "f4c94ffb93bca15bb822d2ebd371e607"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.573094 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "d77a2d1431e44e179010a04cb7a59b7f"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "c44c4595b085e6039f80df0525c41ceb"
server_key_iv: "f4c94ffb93bca15bb822d2ebd371e607"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.574304 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:09.574929 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.575076 22705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:09.575135 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "d77a2d1431e44e179010a04cb7a59b7f"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "c44c4595b085e6039f80df0525c41ceb"
server_key_iv: "f4c94ffb93bca15bb822d2ebd371e607"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.575181 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:09.591790 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:09.593544 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:09.593690 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:09.593935 22705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:09.594192 22705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:09.594238 22705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.594266 22705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:09.594293 22705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.599207 22705 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39005
I20260925 05:06:09.599253 22833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39005 every 8 connection(s)
I20260925 05:06:09.599536 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:09.603902 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22705
I20260925 05:06:09.603996 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:09.604147 14468 external_mini_cluster.cc:1428] Setting key ee666fbf9aafcc29b5aaf52f0fee36c1
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:09.608735 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.600199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49487 (local address 127.14.33.62:42263)
0925 05:06:09.600240 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:09.600242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:09.600433 (+ 191us) server_negotiation.cc:407] Connection header received
0925 05:06:09.600861 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:09.600863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:09.600877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:09.600931 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:09.601830 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.602297 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.603230 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.603426 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.603494 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.606111 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:09.606112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:09.606113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:09.606120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:09.607414 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.607858 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.607858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.607859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.607904 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.608150 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.608150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.608150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.608234 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:09.608284 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:09.608613 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 05:06:09.608675 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:06:09.609009 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.600170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:49487)
0925 05:06:09.600322 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:09.600353 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:09.600769 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:09.601011 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:09.601015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:09.601564 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:09.601736 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:09.601740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.602384 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.602385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:09.603143 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:09.603145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.603500 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.603501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:09.603619 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.604067 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:09.604076 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:09.606023 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:09.607491 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:09.607498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:09.607499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:09.607783 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:09.607959 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:09.607959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:09.607960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:09.608085 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:09.608296 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:09.608302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:09.608474 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 05:06:09.608601 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 05:06:09.608768 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260925 05:06:09.609504 22834 heartbeater.cc:347] Connected to a master server at 127.14.33.62:42263
I20260925 05:06:09.609596 22834 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:09.609869 22834 heartbeater.cc:511] Master 127.14.33.62:42263 requested a full tablet report, sending...
I20260925 05:06:09.610356 22643 ts_manager.cc:194] Registered new tserver with Master: d77a2d1431e44e179010a04cb7a59b7f (127.14.33.1:39005)
I20260925 05:06:09.611080 22643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:49487
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:06:09.629590 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:42263
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:09.703593 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:09.703770 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:09.703790 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:09.704838 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:09.704874 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:09.704917 22839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:09.706259 22839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:09.706473 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:09.706684 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:09.711247 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.711270 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.711256 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:09.711308 22839 server_base.cc:1034] running on GCE node
I20260925 05:06:09.711681 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:09.711897 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:09.713022 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790312769713007 us; error 32 us; skew 500 ppm
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:09.715605 22839 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:06:09.716058 22839 webserver.cc:460] Webserver started at http://127.14.33.2:45677/ using document root <none> and password file <none>
I20260925 05:06:09.716252 22839 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:09.716308 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:09.716425 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:09.717422 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "bee72c990db24203af644c9a775561f0"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "76012cb0c8a389901139f48f3e99eec0"
server_key_iv: "07b0c624a26416c01d90c9cbf4938060"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.717760 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "bee72c990db24203af644c9a775561f0"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "76012cb0c8a389901139f48f3e99eec0"
server_key_iv: "07b0c624a26416c01d90c9cbf4938060"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.718885 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:06:09.719511 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.719671 22839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:09.719765 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "bee72c990db24203af644c9a775561f0"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "76012cb0c8a389901139f48f3e99eec0"
server_key_iv: "07b0c624a26416c01d90c9cbf4938060"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.719836 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:09.733381 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:09.735039 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:09.735193 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:09.735396 22839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:09.735680 22839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:09.735729 22839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.735759 22839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:09.735788 22839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.741511 22839 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:37139
I20260925 05:06:09.741555 22967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:37139 every 8 connection(s)
I20260925 05:06:09.741842 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:09.744050 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22839
I20260925 05:06:09.744177 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:09.744349 14468 external_mini_cluster.cc:1428] Setting key 5c2b069ae289a3ba3b13dea514b3c4ea
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:09.750810 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.742449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:59679 (local address 127.14.33.62:42263)
0925 05:06:09.742492 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:09.742494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:09.742693 (+ 199us) server_negotiation.cc:407] Connection header received
0925 05:06:09.743118 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:09.743119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:09.743128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:09.743171 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:09.744045 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.744746 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.746032 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.746190 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.746230 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.748134 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:09.748135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:09.748136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:09.748143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:09.749628 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.749964 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.749966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.749966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.750027 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.750229 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.750230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.750230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.750368 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:09.750433 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:09.750625 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:06:09.750719 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:06:09.750831 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.742435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:59679)
0925 05:06:09.742573 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:09.742602 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:09.743036 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:09.743225 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:09.743227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:09.743731 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:09.743957 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:09.743964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.744821 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.744822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:09.745938 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:09.745942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.746253 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.746254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:09.746378 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.746867 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:09.746874 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:09.748051 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:09.749702 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:09.749707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:09.749708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:09.749897 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:09.750091 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:09.750091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:09.750091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:09.750173 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:09.750428 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:09.750431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:09.750518 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 05:06:09.750584 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 05:06:09.750675 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260925 05:06:09.751330 22968 heartbeater.cc:347] Connected to a master server at 127.14.33.62:42263
I20260925 05:06:09.751425 22968 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:09.751569 22968 heartbeater.cc:511] Master 127.14.33.62:42263 requested a full tablet report, sending...
I20260925 05:06:09.751920 22643 ts_manager.cc:194] Registered new tserver with Master: bee72c990db24203af644c9a775561f0 (127.14.33.2:37139)
I20260925 05:06:09.752269 22643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:59679
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:06:09.767750 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:42263
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:09.845578 22973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:09.845759 22973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:09.845790 22973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:09.846868 22973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:09.846938 22973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:09.846974 22973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:09.848193 22973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:09.848412 22973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:09.848587 22973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:06:09.852910 22973 server_base.cc:1034] running on GCE node
W20260925 05:06:09.852897 22979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.852907 22981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:09.852895 22978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:09.853282 22973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:09.853447 22973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:09.854601 22973 hybrid_clock.cc:648] HybridClock initialized: now 1790312769854587 us; error 33 us; skew 500 ppm
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:09.856802 22973 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:06:09.857280 22973 webserver.cc:460] Webserver started at http://127.14.33.3:35109/ using document root <none> and password file <none>
I20260925 05:06:09.857496 22973 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:09.857554 22973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:09.857681 22973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:09.858565 22973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "673dc4237697408ba29ac5053b576494"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "5720ad38df4eaeee067d191bd48363ea"
server_key_iv: "f982bebfd8e7b4258417063b10e9fde0"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.858886 22973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "673dc4237697408ba29ac5053b576494"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "5720ad38df4eaeee067d191bd48363ea"
server_key_iv: "f982bebfd8e7b4258417063b10e9fde0"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.860009 22973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:06:09.860616 22988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.860772 22973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:09.860857 22973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "673dc4237697408ba29ac5053b576494"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "5720ad38df4eaeee067d191bd48363ea"
server_key_iv: "f982bebfd8e7b4258417063b10e9fde0"
server_key_version: "encryptionkey@0"
I20260925 05:06:09.860922 22973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:09.866626 22973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:09.868124 22973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:09.868256 22973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:09.868433 22973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:09.868690 22973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:09.868736 22973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.868770 22973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:09.868790 22973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:09.874156 22973 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:45019
I20260925 05:06:09.874203 23101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:45019 every 8 connection(s)
I20260925 05:06:09.874440 22973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:06:09 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312769, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:09.881496 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 22973
I20260925 05:06:09.881618 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:09.881798 14468 external_mini_cluster.cc:1428] Setting key 7d0a8712f56484c42c573331fea949c0
I20260925 05:06:09.882359 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.874987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:42073 (local address 127.14.33.62:42263)
0925 05:06:09.875041 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:09.875043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:09.875280 (+ 237us) server_negotiation.cc:407] Connection header received
0925 05:06:09.875690 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:09.875691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:09.875704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:09.875756 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:09.876541 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.877001 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.877968 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.878153 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.878224 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.879977 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:09.879978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:09.879979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:09.879986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:09.880997 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.881420 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.881421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.881421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.881474 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:09.881744 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:09.881745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:09.881745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:09.881869 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:09.881938 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:09.882185 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:06:09.882275 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:06:09.882484 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:09.874970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:42073)
0925 05:06:09.875136 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:09.875194 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:09.875578 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:09.875816 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:09.875820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:09.876286 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:09.876436 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:09.876439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.877092 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.877093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:09.877894 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:09.877896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:09.878240 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:09.878241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:09.878358 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:09.878718 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:09.878725 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:09.879894 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:09.881086 (+ 1192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:09.881092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:09.881094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:09.881351 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:09.881539 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:09.881539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:09.881540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:09.881670 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:09.881940 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:09.881946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:09.882068 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 05:06:09.882164 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:06:09.882294 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260925 05:06:09.882879 23102 heartbeater.cc:347] Connected to a master server at 127.14.33.62:42263
I20260925 05:06:09.882970 23102 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:09.883122 23102 heartbeater.cc:511] Master 127.14.33.62:42263 requested a full tablet report, sending...
I20260925 05:06:09.883431 22643 ts_manager.cc:194] Registered new tserver with Master: 673dc4237697408ba29ac5053b576494 (127.14.33.3:45019)
I20260925 05:06:09.883781 22643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:42073
I20260925 05:06:09.893618 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab.
Entry for principal oryx/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab.
WARNING: no policy specified for oryx/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab.
Entry for principal oryx/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab.
WARNING: no policy specified for oryx/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab.
Entry for principal oryx/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab.
I20260925 05:06:09.951001 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22705
I20260925 05:06:09.955415 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22839
I20260925 05:06:09.959118 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22973
I20260925 05:06:09.963160 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 22611
I20260925 05:06:09.968017 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42263
--webserver_interface=127.14.33.62
--webserver_port=39979
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:10.043047 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:10.043228 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:10.043248 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:10.044384 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:10.044423 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:10.044436 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:10.044448 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:10.044494 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:10.045874 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=39979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:10.046099 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:10.046315 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:06:10.050647 23111 server_base.cc:1034] running on GCE node
W20260925 05:06:10.050618 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:10.050611 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:10.050618 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.050992 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:10.051312 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:10.052459 23111 hybrid_clock.cc:648] HybridClock initialized: now 1790312770052452 us; error 39 us; skew 500 ppm
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312770, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:10.054724 23111 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:06:10.055104 23111 webserver.cc:460] Webserver started at http://127.14.33.62:39979/ using document root <none> and password file <none>
I20260925 05:06:10.055328 23111 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:10.055403 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:10.056674 23111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:10.057329 23126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.057483 23111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:10.057570 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "95b604d5c3c64cc191edf0f0ff9b9f04"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "89ee67d39f4b9e35c87993cc602b3ee0"
server_key_iv: "3b2e53d277b2af2f6071717c540f3e0e"
server_key_version: "encryptionkey@0"
I20260925 05:06:10.057894 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:10.087278 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:10.089363 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:10.089509 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:10.093086 23111 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:42263
I20260925 05:06:10.093173 23178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:42263 every 8 connection(s)
I20260925 05:06:10.093502 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:10.094810 23179 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:06:10.095449 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Bootstrap starting.
I20260925 05:06:10.102425 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23111
I20260925 05:06:10.102700 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:39005
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:42263
--webserver_port=39593
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:06:10.103408 23179 log.cc:824] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:10.105464 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:06:10.105741 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Bootstrap complete.
I20260925 05:06:10.107874 23179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:10.108093 23179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b604d5c3c64cc191edf0f0ff9b9f04, State: Initialized, Role: FOLLOWER
I20260925 05:06:10.108261 23179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:10.108341 23179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:10.108387 23179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:10.108428 23179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:06:10.109376 23179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:10.109480 23179 leader_election.cc:302] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b604d5c3c64cc191edf0f0ff9b9f04; no voters:
I20260925 05:06:10.109643 23179 leader_election.cc:288] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:06:10.109673 23183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:06:10.109784 23183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [term 2 LEADER]: Becoming Leader. State: Replica: 95b604d5c3c64cc191edf0f0ff9b9f04, State: Running, Role: LEADER
I20260925 05:06:10.109876 23179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:10.109860 23183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } }
I20260925 05:06:10.110100 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b604d5c3c64cc191edf0f0ff9b9f04. Latest consensus state: current_term: 2 leader_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } } }
I20260925 05:06:10.110169 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:10.110105 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b604d5c3c64cc191edf0f0ff9b9f04" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42263 } } }
I20260925 05:06:10.110214 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:10.110421 23187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:10.110972 23187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:10.111311 23187 catalog_manager.cc:1259] Loaded cluster ID: bf0bd01a4bbc46d5a49420886d36ba66
I20260925 05:06:10.111400 23187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:10.112757 23187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:10.112905 23187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95b604d5c3c64cc191edf0f0ff9b9f04: Loaded TSK: 0
I20260925 05:06:10.113090 23187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:06:10.177655 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:10.177834 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:10.177857 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:10.179008 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:10.179049 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:10.179075 23182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:10.180426 23182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.1.keytab
--principal=oryx/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:39005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=39593
--tserver_master_addrs=127.14.33.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:10.180625 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:10.180841 23182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:10.185329 23208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.185465 23182 server_base.cc:1034] running on GCE node
W20260925 05:06:10.185644 23205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:10.185832 23206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.186045 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:10.186307 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:10.187464 23182 hybrid_clock.cc:648] HybridClock initialized: now 1790312770187437 us; error 39 us; skew 500 ppm
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312770, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:10.190665 23182 init.cc:377] Logged in from keytab as oryx/127.14.33.1@KRBTEST.COM (short username oryx)
I20260925 05:06:10.191119 23182 webserver.cc:460] Webserver started at http://127.14.33.1:39593/ using document root <none> and password file <none>
I20260925 05:06:10.191383 23182 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:10.191452 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:10.192936 23182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:06:10.193598 23215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.193742 23182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.193816 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "d77a2d1431e44e179010a04cb7a59b7f"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "c44c4595b085e6039f80df0525c41ceb"
server_key_iv: "f4c94ffb93bca15bb822d2ebd371e607"
server_key_version: "encryptionkey@0"
I20260925 05:06:10.194103 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:10.213263 23182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:10.215179 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:10.215335 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:10.215545 23182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:10.215829 23182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:10.215880 23182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.215919 23182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:10.215947 23182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.222844 23182 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39005
I20260925 05:06:10.222911 23328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39005 every 8 connection(s)
I20260925 05:06:10.223219 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:10.227609 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23182
I20260925 05:06:10.227890 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:37139
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:42263
--webserver_port=45677
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:06:10.231853 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.224011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:58247 (local address 127.14.33.62:42263)
0925 05:06:10.224187 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.224194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.224226 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:06:10.224784 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.224791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.224844 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.224982 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.225909 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.227110 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.228182 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.228575 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.228680 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.231731 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:58247: BlockingRecv error: recv got EOF from 127.14.33.1:58247 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260925 05:06:10.231868 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.223948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:58247)
0925 05:06:10.224112 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.224150 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.224593 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.225022 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.225026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.225610 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.225796 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.225800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.227192 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.227192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.228102 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.228104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.228670 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.228671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.228806 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.229292 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.229300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.231636 (+ 2336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260925 05:06:10.232098 23329 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:42263 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.33.62:42263: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.236958 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.232356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:38877)
0925 05:06:10.232403 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.232413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.232451 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.232574 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.232575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.232742 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.232798 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.232800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.233530 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.233532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.234445 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.234448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.234756 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.234757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.234831 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.235181 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.235187 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.236883 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:06:10.237011 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.232396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:38877 (local address 127.14.33.62:42263)
0925 05:06:10.232442 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.232444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.232457 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:06:10.232513 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.232513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.232523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.232569 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.232872 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.233454 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.234524 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.234690 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.234735 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.236951 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:38877: BlockingRecv error: recv got EOF from 127.14.33.1:38877 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.241150 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.237244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:33075)
0925 05:06:10.237328 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.237339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.237383 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.237524 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.237525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.237636 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.237658 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.237659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.238224 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.238225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.238693 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.238695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.239041 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.239042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.239092 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.239484 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.239488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.241073 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 05:06:10.241290 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.237291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:33075 (local address 127.14.33.62:42263)
0925 05:06:10.237333 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.237335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.237370 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:06:10.237438 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.237439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.237448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.237508 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.237733 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.238153 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.238786 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.238964 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.239039 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.241199 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:33075: BlockingRecv error: recv got EOF from 127.14.33.1:33075 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 05:06:10.241303 23329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:06:10.308079 23333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:10.308293 23333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:10.308331 23333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:10.309487 23333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:10.309547 23333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:10.309638 23333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:10.311185 23333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.2.keytab
--principal=oryx/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:37139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=45677
--tserver_master_addrs=127.14.33.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:10.311422 23333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:10.311625 23333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:10.316538 23341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.316560 23333 server_base.cc:1034] running on GCE node
W20260925 05:06:10.316545 23339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:10.316550 23338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.316946 23333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:10.317229 23333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:10.318368 23333 hybrid_clock.cc:648] HybridClock initialized: now 1790312770318355 us; error 32 us; skew 500 ppm
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312770, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:10.320621 23333 init.cc:377] Logged in from keytab as oryx/127.14.33.2@KRBTEST.COM (short username oryx)
I20260925 05:06:10.321017 23333 webserver.cc:460] Webserver started at http://127.14.33.2:45677/ using document root <none> and password file <none>
I20260925 05:06:10.321295 23333 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:10.321372 23333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:10.322582 23333 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 05:06:10.323093 23348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.323253 23333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.323333 23333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "bee72c990db24203af644c9a775561f0"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "76012cb0c8a389901139f48f3e99eec0"
server_key_iv: "07b0c624a26416c01d90c9cbf4938060"
server_key_version: "encryptionkey@0"
I20260925 05:06:10.323624 23333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:10.338563 23333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:10.340179 23333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:10.340325 23333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:10.340515 23333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:10.340794 23333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:10.340837 23333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.340873 23333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:10.340900 23333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.346463 23333 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:37139
I20260925 05:06:10.346498 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:37139 every 8 connection(s)
I20260925 05:06:10.346808 23333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.352824 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23333
I20260925 05:06:10.353152 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:45019
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:42263
--webserver_port=35109
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:10.353519 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.347467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:52831 (local address 127.14.33.62:42263)
0925 05:06:10.347537 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.347539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.347691 (+ 152us) server_negotiation.cc:407] Connection header received
0925 05:06:10.348081 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.348082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.348094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.348139 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.348945 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.349487 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.350498 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.350679 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.350727 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.353474 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:52831: BlockingRecv error: recv got EOF from 127.14.33.2:52831 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:06:10.353622 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.347432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:52831)
0925 05:06:10.347584 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.347615 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.348006 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.348203 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.348206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.348711 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.348878 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.348881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.349555 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.349556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.350420 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.350422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.350740 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.350741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.350828 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.351204 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.351211 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.353411 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260925 05:06:10.353824 23462 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:42263 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.33.62:42263: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.358697 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.354008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:44413)
0925 05:06:10.354121 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.354131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.354172 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.354318 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.354319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.354470 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.354523 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.354525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.355376 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.355377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.356091 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.356094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.356404 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.356405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.356474 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.356798 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.356803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.358613 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
W20260925 05:06:10.358723 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.354055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:44413 (local address 127.14.33.62:42263)
0925 05:06:10.354092 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.354093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.354161 (+ 68us) server_negotiation.cc:407] Connection header received
0925 05:06:10.354241 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.354241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.354249 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.354296 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.354598 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.355301 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.356165 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.356336 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.356386 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.358684 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:44413: BlockingRecv error: recv got EOF from 127.14.33.2:44413 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.363404 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.358925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:56195)
0925 05:06:10.358971 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.358980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.359015 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.359169 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.359170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.359325 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.359360 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.359361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.359920 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.359920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.360814 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.360816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.361156 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.361158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.361230 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.361583 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.361588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.363321 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:06:10.363420 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.358982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:56195 (local address 127.14.33.62:42263)
0925 05:06:10.359018 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.359020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.359026 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:06:10.359089 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.359089 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.359097 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.359143 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.359428 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.359856 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.360884 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.361046 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.361093 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.363385 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:56195: BlockingRecv error: recv got EOF from 127.14.33.2:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 05:06:10.363509 23462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:06:10.434409 23466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:10.434593 23466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:10.434614 23466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:10.435742 23466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:10.435783 23466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:10.435832 23466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:10.437288 23466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/krb5kdc/oryx_127.14.33.3.keytab
--principal=oryx/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:45019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=35109
--tserver_master_addrs=127.14.33.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:10.437501 23466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:10.437704 23466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:10.442327 23472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.442425 23466 server_base.cc:1034] running on GCE node
W20260925 05:06:10.442348 23474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:10.442350 23471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:10.442718 23466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:10.442942 23466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:10.444106 23466 hybrid_clock.cc:648] HybridClock initialized: now 1790312770444098 us; error 29 us; skew 500 ppm
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312770, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:10.446316 23466 init.cc:377] Logged in from keytab as oryx/127.14.33.3@KRBTEST.COM (short username oryx)
I20260925 05:06:10.446713 23466 webserver.cc:460] Webserver started at http://127.14.33.3:35109/ using document root <none> and password file <none>
I20260925 05:06:10.446928 23466 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:10.447002 23466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:10.448280 23466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:10.448729 23481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.448863 23466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.448951 23466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "673dc4237697408ba29ac5053b576494"
format_stamp: "Formatted at 2026-09-25 05:06:09 on dist-test-slave-4dmq"
server_key: "5720ad38df4eaeee067d191bd48363ea"
server_key_iv: "f982bebfd8e7b4258417063b10e9fde0"
server_key_version: "encryptionkey@0"
I20260925 05:06:10.449288 23466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:10.467784 23466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:10.469463 23466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:10.469605 23466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:10.469800 23466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:10.470067 23466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:10.470119 23466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.470157 23466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:10.470183 23466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:10.475829 23466 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:45019
I20260925 05:06:10.475865 23594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:45019 every 8 connection(s)
I20260925 05:06:10.476173 23466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:10.478658 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23466
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:06:10.483271 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.476837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:33661 (local address 127.14.33.62:42263)
0925 05:06:10.476900 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.476902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.477079 (+ 177us) server_negotiation.cc:407] Connection header received
0925 05:06:10.477573 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.477574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.477587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.477647 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.478529 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.479231 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.480135 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.480336 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.480414 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.483212 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:33661: BlockingRecv error: recv got EOF from 127.14.33.3:33661 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:06:10.483340 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.476835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:33661)
0925 05:06:10.476970 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.477000 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.477472 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.477710 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.477718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.478273 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.478437 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.478441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.479307 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.479308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.480049 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.480051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.480425 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.480426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.480536 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.481019 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.481027 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.483152 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
W20260925 05:06:10.483544 23595 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:42263 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.33.62:42263: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
I20260925 05:06:10.485847 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.479108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.14.33.62:42263)
0925 05:06:10.479228 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.479230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.479237 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:10.479263 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.479264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.479272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.479343 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.479903 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.480360 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.481101 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.481331 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.481402 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.482628 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:10.482633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:10.482633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:10.482644 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:10.484473 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:10.484823 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:10.484826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:10.484826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:10.484938 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:10.485164 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:10.485165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:10.485165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:10.485374 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:10.485509 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:10.485635 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:06:10.485742 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.487722 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.483777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:48907)
0925 05:06:10.483816 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.483824 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.483889 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.484007 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.484008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.484118 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.484154 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.484155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.484742 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.484743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.485285 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.485286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.485613 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.485614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.485698 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.486083 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.486088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.487685 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 05:06:10.487840 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.483821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:48907 (local address 127.14.33.62:42263)
0925 05:06:10.483883 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.483885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.483893 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:10.483942 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.483942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.483951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.484004 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.484236 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.484654 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.485355 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.485538 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.485611 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.487774 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:48907: BlockingRecv error: recv got EOF from 127.14.33.3:48907 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:10 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:10.491631 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:10.487959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:35647)
0925 05:06:10.488025 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:10.488034 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:10.488067 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:10.488208 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:10.488209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:10.488330 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:10.488351 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.488352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.488921 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.488922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.489481 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:10.489483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.489823 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.489824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:10.489873 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.490175 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:10.490179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:10.491562 (+ 1383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 05:06:10.491677 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:10.488006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:35647 (local address 127.14.33.62:42263)
0925 05:06:10.488064 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:10.488066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:10.488074 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:10.488125 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:10.488126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:10.488134 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:10.488204 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:10.488426 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.488851 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.489562 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:10.489744 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:10.489822 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:10.491621 (+ 1799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:35647: BlockingRecv error: recv got EOF from 127.14.33.3:35647 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 05:06:10.491726 23595 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 05:06:11 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:11 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:11.245992 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:11.241664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:53945)
0925 05:06:11.241805 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:11.241820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:11.241864 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:11.241998 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:11.241998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:11.242143 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:11.242191 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:11.242192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.242796 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.242797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:11.243353 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:11.243355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.243662 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.243663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:11.243704 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:11.244124 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:11.244128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:11.245930 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
W20260925 05:06:11.246091 23601 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:11.241718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:53945 (local address 127.14.33.62:42263)
0925 05:06:11.241883 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:11.241884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:11.241892 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:11.241912 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:11.241913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:11.241922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:11.241984 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:11.242250 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.242730 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.243416 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.243608 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.243654 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:11.246025 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:53945: BlockingRecv error: recv got EOF from 127.14.33.1:53945 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":117,"threads_started":1}
Sep 25 05:06:11 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:11 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:11.367812 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:11.363837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:33317)
0925 05:06:11.364007 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:11.364016 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:11.364064 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:11.364160 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:11.364161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:11.364311 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:11.364359 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:11.364360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.364913 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.364913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:11.365450 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:11.365451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.365741 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.365742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:11.365779 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:11.366191 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:11.366195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:11.367750 (+ 1555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
W20260925 05:06:11.367918 23601 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:11.363911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:33317 (local address 127.14.33.62:42263)
0925 05:06:11.363982 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:11.363984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:11.364066 (+ 82us) server_negotiation.cc:407] Connection header received
0925 05:06:11.364103 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:11.364104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:11.364112 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:11.364155 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:11.364413 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.364856 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.365512 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.365685 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.365733 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:11.367843 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:33317: BlockingRecv error: recv got EOF from 127.14.33.2:33317 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 05:06:11 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:11 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:11.496563 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:11.492076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:41197)
0925 05:06:11.492187 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:11.492202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:11.492255 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:11.492387 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:11.492388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:11.492599 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:11.492657 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:11.492659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.493595 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.493596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:11.494104 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:11.494106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.494439 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.494440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:11.494476 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:11.494939 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:11.494944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:11.496502 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260925 05:06:11.496647 23601 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:11.492129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:41197 (local address 127.14.33.62:42263)
0925 05:06:11.492174 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:11.492176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:11.492256 (+ 80us) server_negotiation.cc:407] Connection header received
0925 05:06:11.492297 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:11.492298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:11.492309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:11.492382 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:11.492741 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.493511 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.494171 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:11.494372 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:11.494431 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:11.496585 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:41197: BlockingRecv error: recv got EOF from 127.14.33.3:41197 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 05:06:12 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:12 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:12.250720 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:12.246452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:52227)
0925 05:06:12.246567 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:12.246583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:12.246629 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:12.246752 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:12.246752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:12.246895 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:12.246943 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:12.246945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.247539 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.247540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:12.248048 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:12.248050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.248362 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.248363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:12.248422 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:12.248810 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:12.248814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:12.250631 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
W20260925 05:06:12.250785 23605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:12.246498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:52227 (local address 127.14.33.62:42263)
0925 05:06:12.246623 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:12.246625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:12.246633 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:12.246675 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:12.246675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:12.246685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:12.246740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:12.246996 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.247479 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.248105 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.248273 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.248317 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:12.250706 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:52227: BlockingRecv error: recv got EOF from 127.14.33.1:52227 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
Sep 25 05:06:12 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:12 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:12.372402 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:12.368277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:57809)
0925 05:06:12.368419 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:12.368428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:12.368479 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:12.368581 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:12.368582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:12.368740 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:12.368786 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:12.368788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.369405 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.369406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:12.369942 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:12.369943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.370232 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.370233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:12.370273 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:12.370703 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:12.370707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:12.372346 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
W20260925 05:06:12.372547 23605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:12.368329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:57809 (local address 127.14.33.62:42263)
0925 05:06:12.368391 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:12.368393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:12.368473 (+ 80us) server_negotiation.cc:407] Connection header received
0925 05:06:12.368520 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:12.368521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:12.368530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:12.368577 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:12.368841 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.369335 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.370003 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.370175 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.370223 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:12.372450 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:57809: BlockingRecv error: recv got EOF from 127.14.33.2:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 05:06:12 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:12 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:12.501289 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:12.496991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:58085)
0925 05:06:12.497167 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:12.497180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:12.497238 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:12.497353 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:12.497353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:12.497506 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:12.497552 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:12.497553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.498133 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.498134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:12.498627 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:12.498628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.498911 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.498912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:12.498951 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:12.499415 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:12.499420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:12.501232 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
W20260925 05:06:12.501339 23605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:12.497137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:58085 (local address 127.14.33.62:42263)
0925 05:06:12.497203 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:12.497205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:12.497224 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:06:12.497279 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:12.497279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:12.497291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:12.497337 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:12.497611 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.498074 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.498695 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:12.498858 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:12.498902 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:12.501298 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:58085: BlockingRecv error: recv got EOF from 127.14.33.3:58085 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 05:06:13 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:13 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:13.255415 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:13.251122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:49041)
0925 05:06:13.251254 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:13.251263 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:13.251314 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:13.251457 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:13.251458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:13.251603 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:13.251651 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:13.251652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.252245 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.252246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:13.252830 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:13.252833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.253164 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.253165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:13.253229 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.253708 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:13.253713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:13.255313 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260925 05:06:13.255458 23609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:13.251197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49041 (local address 127.14.33.62:42263)
0925 05:06:13.251297 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:13.251303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:13.251312 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:13.251369 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:13.251369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:13.251380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:13.251436 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:13.251706 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.252183 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.252891 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.253073 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.253172 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.255390 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:49041: BlockingRecv error: recv got EOF from 127.14.33.1:49041 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":55,"threads_started":1}
Sep 25 05:06:13 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:13 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:13.377180 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:13.372831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:34621)
0925 05:06:13.372996 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:13.373007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:13.373069 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:13.373194 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:13.373195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:13.373353 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:13.373402 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:13.373403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.374020 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.374020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:13.374564 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:13.374565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.374872 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.374873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:13.374911 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.375386 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:13.375392 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:13.377047 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
W20260925 05:06:13.377251 23609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:13.372892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:34621 (local address 127.14.33.62:42263)
0925 05:06:13.372962 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:13.372964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:13.373068 (+ 104us) server_negotiation.cc:407] Connection header received
0925 05:06:13.373130 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:13.373130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:13.373141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:13.373190 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:13.373457 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.373954 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.374630 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.374814 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.374862 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.377203 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:34621: BlockingRecv error: recv got EOF from 127.14.33.2:34621 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 05:06:13 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:13 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:13.506002 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:13.501762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:45727)
0925 05:06:13.501937 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:13.501948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:13.502004 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:13.502128 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:13.502129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:13.502283 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:13.502328 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:13.502329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.502947 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.502948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:13.503472 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:13.503474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.503770 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.503771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:13.503805 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.504229 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:13.504233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:13.505936 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
W20260925 05:06:13.506250 23609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:13.501800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:45727 (local address 127.14.33.62:42263)
0925 05:06:13.501870 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:13.501872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:13.502006 (+ 134us) server_negotiation.cc:407] Connection header received
0925 05:06:13.502051 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:13.502051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:13.502064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:13.502113 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:13.502383 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.502879 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.503534 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:13.503704 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:13.503757 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:13.506034 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:45727: BlockingRecv error: recv got EOF from 127.14.33.3:45727 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 05:06:14 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:14 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:14.260254 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:14.255932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:49775)
0925 05:06:14.256067 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:14.256083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:14.256149 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:14.256312 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:14.256313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:14.256460 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:14.256509 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:14.256511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.257210 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.257211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:14.257736 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:14.257738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.258066 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.258066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:14.258103 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:14.258519 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:14.258523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:14.260192 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260925 05:06:14.260360 23613 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:14.255991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49775 (local address 127.14.33.62:42263)
0925 05:06:14.256133 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:14.256135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:14.256143 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:14.256203 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:14.256204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:14.256213 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:14.256273 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:14.256574 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.257080 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.257808 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.258003 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.258058 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:14.260283 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:49775: BlockingRecv error: recv got EOF from 127.14.33.1:49775 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
Sep 25 05:06:14 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:14 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:14.382190 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:14.377794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:41203)
0925 05:06:14.377937 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:14.377948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:14.378010 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:14.378138 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:14.378138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:14.378299 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:14.378348 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:14.378350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.378945 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.378946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:14.379462 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:14.379464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.379756 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.379757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:14.379794 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:14.380279 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:14.380285 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:14.382130 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260925 05:06:14.382275 23613 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:14.377869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:41203 (local address 127.14.33.62:42263)
0925 05:06:14.377945 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:14.377947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:14.377998 (+ 51us) server_negotiation.cc:407] Connection header received
0925 05:06:14.378055 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:14.378055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:14.378067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:14.378122 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:14.378409 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.378886 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.379528 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.379701 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.379753 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:14.382213 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:41203: BlockingRecv error: recv got EOF from 127.14.33.2:41203 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 05:06:14 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:14 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:14.511205 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:14.506491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:34981)
0925 05:06:14.506630 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:14.506641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:14.506701 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:14.506832 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:14.506832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:14.507006 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:14.507055 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:14.507056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.507706 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.507707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:14.508223 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:14.508225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.508581 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.508582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:14.508642 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:14.509167 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:14.509172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:14.511150 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260925 05:06:14.511312 23613 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:14.506554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:34981 (local address 127.14.33.62:42263)
0925 05:06:14.506616 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:14.506618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:14.506703 (+ 85us) server_negotiation.cc:407] Connection header received
0925 05:06:14.506752 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:14.506752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:14.506764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:14.506818 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:14.507115 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.507645 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.508290 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:14.508512 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:14.508586 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:14.511226 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:34981: BlockingRecv error: recv got EOF from 127.14.33.3:34981 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 05:06:15 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:15 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:15.272653 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:15.260645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:40529)
0925 05:06:15.260791 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:15.260803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:15.260860 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:15.261024 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:15.261024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:15.261216 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:15.261265 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:15.261267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.261868 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.261868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:15.262356 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:15.262357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.262647 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.262647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:15.262683 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.270885 (+ 8202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:15.270890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:15.272599 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
W20260925 05:06:15.272771 23617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:15.260713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:40529 (local address 127.14.33.62:42263)
0925 05:06:15.260841 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:15.260843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:15.260851 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:15.260933 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:15.260934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:15.260947 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:15.261012 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:15.261329 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.261807 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.262415 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.262593 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.262640 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.272690 (+ 10050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:40529: BlockingRecv error: recv got EOF from 127.14.33.1:40529 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
Sep 25 05:06:15 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:15 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:15.387217 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:15.382709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:41325)
0925 05:06:15.382869 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:15.382883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:15.382948 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:15.383099 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:15.383100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:15.383256 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:15.383302 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:15.383304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.383903 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.383904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:15.384404 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:15.384405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.384789 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.384789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:15.384827 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.385326 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:15.385331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:15.387113 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
W20260925 05:06:15.387279 23617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:15.382821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:41325 (local address 127.14.33.62:42263)
0925 05:06:15.382886 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:15.382888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:15.382940 (+ 52us) server_negotiation.cc:407] Connection header received
0925 05:06:15.383019 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:15.383019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:15.383031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:15.383089 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:15.383362 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.383834 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.384488 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.384717 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.384778 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.387214 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:41325: BlockingRecv error: recv got EOF from 127.14.33.2:41325 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 05:06:15 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:15 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:15.515908 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:15.511650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:45745)
0925 05:06:15.511812 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:15.511823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:15.511904 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:15.512055 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:15.512056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:15.512220 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:15.512264 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:15.512265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.512914 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.512915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:15.513448 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:15.513450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.513796 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.513797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:15.513852 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.514279 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:15.514283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:15.515821 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
W20260925 05:06:15.515962 23617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:15.511721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:45745 (local address 127.14.33.62:42263)
0925 05:06:15.511789 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:15.511791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:15.511888 (+ 97us) server_negotiation.cc:407] Connection header received
0925 05:06:15.511959 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:15.511959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:15.511974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:15.512034 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:15.512338 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.512838 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.513527 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:15.513711 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:15.513794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:15.515898 (+ 2104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:45745: BlockingRecv error: recv got EOF from 127.14.33.3:45745 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 05:06:16 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:16 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:16.277875 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:16.273168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:44479)
0925 05:06:16.273323 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:16.273335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:16.273408 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:16.273545 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:16.273546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:16.273754 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:16.273804 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:16.273806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.274453 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.274453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:16.274975 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:16.274977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.275308 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.275309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:16.275352 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:16.275899 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:16.275904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:16.277818 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
W20260925 05:06:16.278003 23621 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:16.273210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:44479 (local address 127.14.33.62:42263)
0925 05:06:16.273333 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:16.273335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:16.273400 (+ 65us) server_negotiation.cc:407] Connection header received
0925 05:06:16.273456 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:16.273456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:16.273467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:16.273525 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:16.273884 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.274368 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.275050 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.275237 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.275293 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:16.277897 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:44479: BlockingRecv error: recv got EOF from 127.14.33.1:44479 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
Sep 25 05:06:16 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:16 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 05:06:16.393996 23621 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:16.387714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:60103 (local address 127.14.33.62:42263)
0925 05:06:16.387781 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:16.387783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:16.387901 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:06:16.388015 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:16.388016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:16.388031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:16.388123 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:16.388543 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.389455 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.390581 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.390788 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.390831 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:16.393950 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:60103: BlockingRecv error: recv got EOF from 127.14.33.2:60103 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:06:16.393990 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:16.387690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:60103)
0925 05:06:16.387837 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:16.387848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:16.387913 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:16.388168 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:16.388176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:16.388395 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:16.388457 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:16.388459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.389550 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.389551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:16.390494 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:16.390496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.390934 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.390935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:16.390994 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:16.391483 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:16.391488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:16.393873 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
Sep 25 05:06:16 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:16 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:16.520983 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:16.516388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:56147)
0925 05:06:16.516525 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:16.516536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:16.516617 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:16.516741 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:16.516742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:16.516915 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:16.516959 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:16.516961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.517695 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.517696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:16.518240 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:16.518242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.518618 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.518619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:16.518695 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:16.519198 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:16.519203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:16.520929 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20260925 05:06:16.521112 23621 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:16.516442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:56147 (local address 127.14.33.62:42263)
0925 05:06:16.516511 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:16.516513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:16.516600 (+ 87us) server_negotiation.cc:407] Connection header received
0925 05:06:16.516670 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:16.516671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:16.516686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:16.516737 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:16.517036 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.517593 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.518323 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:16.518532 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:16.518613 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:16.521015 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:56147: BlockingRecv error: recv got EOF from 127.14.33.3:56147 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 05:06:17 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:17 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:17.282833 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:17.278304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:60487)
0925 05:06:17.278434 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:17.278450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:17.278504 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:17.278676 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:17.278677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:17.278828 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:17.278877 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:17.278879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.279502 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.279503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:17.280015 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:17.280017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.280370 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.280370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:17.280410 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:17.280841 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:17.280846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:17.282774 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260925 05:06:17.282954 23625 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:17.278418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:60487 (local address 127.14.33.62:42263)
0925 05:06:17.278542 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:17.278544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:17.278553 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:17.278584 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:17.278584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:17.278594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:17.278659 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:17.278944 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.279437 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.280094 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.280298 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.280360 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:17.282870 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:60487: BlockingRecv error: recv got EOF from 127.14.33.1:60487 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
Sep 25 05:06:17 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:17 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:17.399165 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:17.394554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:50639)
0925 05:06:17.394684 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:17.394698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:17.394761 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:17.394890 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:17.394891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:17.395062 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:17.395108 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:17.395109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.395761 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.395762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:17.396296 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:17.396297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.396710 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.396711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:17.396771 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:17.397280 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:17.397285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:17.399078 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
W20260925 05:06:17.399210 23625 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:17.394664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:50639 (local address 127.14.33.62:42263)
0925 05:06:17.394718 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:17.394720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:17.394756 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:06:17.394818 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:17.394819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:17.394831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:17.394877 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:17.395185 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.395677 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.396380 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.396627 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.396700 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:17.399153 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:50639: BlockingRecv error: recv got EOF from 127.14.33.2:50639 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 05:06:17 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:17 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:17.526432 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:17.521518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:44367)
0925 05:06:17.521688 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:17.521698 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:17.521776 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:17.521921 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:17.521922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:17.522100 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:17.522145 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:17.522147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.522826 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.522827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:17.523493 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:17.523495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.523891 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.523892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:17.523953 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:17.524444 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:17.524449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:17.526318 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
W20260925 05:06:17.526454 23625 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:17.521574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:44367 (local address 127.14.33.62:42263)
0925 05:06:17.521642 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:17.521645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:17.521760 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:06:17.521831 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:17.521832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:17.521846 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:17.521917 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:17.522233 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.522735 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.523577 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:17.523772 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:17.523891 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:17.526395 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:44367: BlockingRecv error: recv got EOF from 127.14.33.3:44367 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 05:06:18 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:18 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:18.287739 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:18.283310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:54205)
0925 05:06:18.283475 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:18.283486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:18.283539 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:18.283670 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:18.283671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:18.283845 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:18.283907 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:18.283909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.284523 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.284524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:18.285041 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:18.285042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.285399 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.285400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:18.285448 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:18.285950 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:18.285955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:18.287680 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
W20260925 05:06:18.287858 23629 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:18.283429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:54205 (local address 127.14.33.62:42263)
0925 05:06:18.283525 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:18.283527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:18.283536 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:06:18.283594 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:18.283595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:18.283604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:18.283649 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:18.283974 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.284461 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.285107 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.285333 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.285387 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:18.287772 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:54205: BlockingRecv error: recv got EOF from 127.14.33.1:54205 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
Sep 25 05:06:18 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:18 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:18.404255 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:18.399686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:33707)
0925 05:06:18.399820 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:18.399830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:18.399883 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:18.400004 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:18.400004 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:18.400149 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:18.400196 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:18.400197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.400988 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.400991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:18.401625 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:18.401628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.401952 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.401953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:18.401992 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:18.402415 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:18.402419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:18.404146 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260925 05:06:18.404294 23629 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:18.399757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:33707 (local address 127.14.33.62:42263)
0925 05:06:18.399822 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:18.399824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:18.399882 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:06:18.399924 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:18.399925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:18.399935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:18.399990 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:18.400251 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.400872 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.401692 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.401887 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.401942 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:18.404235 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:33707: BlockingRecv error: recv got EOF from 127.14.33.2:33707 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 05:06:18 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:18 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:18.531209 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:18.526858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:33641)
0925 05:06:18.526977 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:18.526988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:18.527042 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:18.527154 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:18.527155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:18.527293 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:18.527338 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:18.527340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.527927 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.527928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:18.528417 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:18.528418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.528686 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.528686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:18.528720 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:18.529268 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:18.529274 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:18.531111 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
W20260925 05:06:18.531262 23629 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:18.526919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:33641 (local address 127.14.33.62:42263)
0925 05:06:18.526971 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:18.526973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:18.527044 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:06:18.527080 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:18.527080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:18.527090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:18.527140 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:18.527401 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.527864 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.528477 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:18.528631 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:18.528676 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:18.531199 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:33641: BlockingRecv error: recv got EOF from 127.14.33.3:33641 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 05:06:19 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:19 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:19.292850 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.288207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:55127)
0925 05:06:19.288395 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.288405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.288456 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.288580 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.288581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.288744 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.288792 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.288793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.289466 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.289467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.289982 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.289983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.290297 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.290297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.290338 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.290746 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.290750 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.292790 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":131,"threads_started":1}
W20260925 05:06:19.292992 23633 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:19.288329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:55127 (local address 127.14.33.62:42263)
0925 05:06:19.288431 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.288432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.288442 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:06:19.288512 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.288512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.288522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.288575 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.288845 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.289383 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.290054 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.290235 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.290281 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.292901 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:55127: BlockingRecv error: recv got EOF from 127.14.33.1:55127 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
Sep 25 05:06:19 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:19 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:19.408959 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.404705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:54091)
0925 05:06:19.404853 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.404871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.404925 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.405027 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.405028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.405191 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.405238 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.405240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.405874 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.405875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.406390 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.406391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.406695 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.406696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.406734 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.407218 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.407223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.408904 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
W20260925 05:06:19.409106 23633 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:19.404783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:54091 (local address 127.14.33.62:42263)
0925 05:06:19.404837 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.404839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.404922 (+ 83us) server_negotiation.cc:407] Connection header received
0925 05:06:19.404965 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.404966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.404977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.405023 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.405300 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.405794 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.406456 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.406637 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.406687 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.409017 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:54091: BlockingRecv error: recv got EOF from 127.14.33.2:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 05:06:19 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:19 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:19.536434 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:19.531673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:53037)
0925 05:06:19.531857 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:19.531868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:19.531950 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:19.532099 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:19.532100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:19.532280 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:19.532323 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.532325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.532999 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.533001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.533537 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:19.533538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.533935 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.533937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:19.533998 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.534455 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:19.534459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:19.536330 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
W20260925 05:06:19.536471 23633 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:19.531756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:53037 (local address 127.14.33.62:42263)
0925 05:06:19.531832 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:19.531834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:19.531935 (+ 101us) server_negotiation.cc:407] Connection header received
0925 05:06:19.532005 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:19.532006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:19.532021 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:19.532091 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:19.532402 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.532894 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.533621 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:19.533849 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:19.533935 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:19.536408 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:53037: BlockingRecv error: recv got EOF from 127.14.33.3:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 05:06:20 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:20 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:20.303113 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.293360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:59499)
0925 05:06:20.293498 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.293516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.293566 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.293797 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.293798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.293972 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.294023 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.294025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.294694 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.294695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.295224 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.295225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.295566 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.295566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.295603 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.301140 (+ 5537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.301145 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.303001 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260925 05:06:20.303158 23637 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.293478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:59499 (local address 127.14.33.62:42263)
0925 05:06:20.293617 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.293619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.293627 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:20.293651 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.293652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.293673 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.293743 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.294097 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.294608 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.295303 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.295499 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.295554 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.303085 (+ 7531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:59499: BlockingRecv error: recv got EOF from 127.14.33.1:59499 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
Sep 25 05:06:20 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:20 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:20.414110 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.409434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:36665)
0925 05:06:20.409595 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.409608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.409698 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.409855 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.409856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.410107 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.410174 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.410177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.410848 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.410850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.411442 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.411444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.411810 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.411811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.411869 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.412361 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.412365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.414049 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
W20260925 05:06:20.414259 23637 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.409490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:36665 (local address 127.14.33.62:42263)
0925 05:06:20.409563 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.409565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.409671 (+ 106us) server_negotiation.cc:407] Connection header received
0925 05:06:20.409766 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.409767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.409779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.409849 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.410254 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.410762 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.411522 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.411727 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.411807 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.414165 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:36665: BlockingRecv error: recv got EOF from 127.14.33.2:36665 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 05:06:20 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:20 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:20.541206 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:20.536898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:35583)
0925 05:06:20.537041 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:20.537052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:20.537111 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:20.537293 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:20.537294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:20.537448 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:20.537497 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.537498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.538112 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.538113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.538634 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:20.538635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.538956 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.538956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:20.538996 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.539413 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:20.539418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:20.541106 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
W20260925 05:06:20.541328 23637 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:20.537021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:35583 (local address 127.14.33.62:42263)
0925 05:06:20.537085 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:20.537086 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:20.537094 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:20.537217 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:20.537217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:20.537228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:20.537275 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:20.537555 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.538049 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.538710 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:20.538893 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:20.538944 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:20.541231 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:35583: BlockingRecv error: recv got EOF from 127.14.33.3:35583 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 05:06:21 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:21 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:21.308211 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.303701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:46091)
0925 05:06:21.303819 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:21.303831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:21.303928 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:21.304132 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:21.304133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:21.304349 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:21.304394 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:21.304395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.305000 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.305001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:21.305603 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:21.305605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.305938 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.305939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:21.305979 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.306463 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:21.306467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:21.308119 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260925 05:06:21.308274 23641 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:21.303735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:46091 (local address 127.14.33.62:42263)
0925 05:06:21.303923 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.303924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.303935 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:06:21.304012 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.304013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.304027 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.304102 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:21.304459 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.304937 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.305663 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.305857 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.305925 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.308218 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:46091: BlockingRecv error: recv got EOF from 127.14.33.1:46091 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":56,"threads_started":1}
Sep 25 05:06:21 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:21 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:21.418788 23642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.414506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:41207)
0925 05:06:21.414660 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:21.414670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:21.414743 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:21.414882 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:21.414883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:21.415046 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:21.415086 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:21.415088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.415667 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.415668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:21.416161 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:21.416162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.416487 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.416488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:21.416544 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.416931 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:21.416935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:21.418681 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
W20260925 05:06:21.418836 23641 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:21.414578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:41207 (local address 127.14.33.62:42263)
0925 05:06:21.414647 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.414650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.414726 (+ 76us) server_negotiation.cc:407] Connection header received
0925 05:06:21.414801 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.414802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.414813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.414875 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:21.415153 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.415594 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.416235 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.416412 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.416485 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.418770 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:41207: BlockingRecv error: recv got EOF from 127.14.33.2:41207 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 05:06:21 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:21 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:21.545785 23643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:21.541646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:45903)
0925 05:06:21.541815 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:21.541824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:21.541877 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:21.541979 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:21.541979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:21.542149 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:21.542193 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:21.542194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.542786 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.542787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:21.543258 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:21.543259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.543543 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.543543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:21.543578 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.544022 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:21.544026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:21.545710 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
W20260925 05:06:21.545846 23641 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:21.541705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:45903 (local address 127.14.33.62:42263)
0925 05:06:21.541797 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:21.541799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:21.541865 (+ 66us) server_negotiation.cc:407] Connection header received
0925 05:06:21.541918 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:21.541919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:21.541930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:21.541973 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:21.542250 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.542727 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.543321 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:21.543485 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:21.543531 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:21.545799 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:45903: BlockingRecv error: recv got EOF from 127.14.33.3:45903 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 05:06:22 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:22 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:22.313424 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:22.308735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:35539)
0925 05:06:22.308892 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:22.308902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:22.308956 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:22.309086 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:22.309087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:22.309329 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:22.309385 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:22.309387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.310159 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.310159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:22.310784 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:22.310786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.311138 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.311139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:22.311190 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:22.311654 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:22.311658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:22.313358 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
W20260925 05:06:22.313512 23645 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:22.308820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:35539 (local address 127.14.33.62:42263)
0925 05:06:22.308939 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:22.308941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:22.308949 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:22.309010 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:22.309011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:22.309020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:22.309067 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:22.309484 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.310096 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.310847 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.311077 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.311127 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:22.313444 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:35539: BlockingRecv error: recv got EOF from 127.14.33.1:35539 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":51,"threads_started":1}
Sep 25 05:06:22 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:22 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:22.424037 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:22.419327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:44919)
0925 05:06:22.419514 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:22.419527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:22.419632 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:22.419792 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:22.419793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:22.419991 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:22.420034 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:22.420035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.420748 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.420750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:22.421311 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:22.421313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.421686 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.421687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:22.421756 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:22.422227 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:22.422231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:22.423981 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
W20260925 05:06:22.424172 23645 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:22.419424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:44919 (local address 127.14.33.62:42263)
0925 05:06:22.419519 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:22.419521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:22.419614 (+ 93us) server_negotiation.cc:407] Connection header received
0925 05:06:22.419709 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:22.419710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:22.419722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:22.419785 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:22.420110 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.420658 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.421396 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.421605 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.421684 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:22.424065 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:44919: BlockingRecv error: recv got EOF from 127.14.33.2:44919 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 05:06:22 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:22 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:22.550762 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:22.546210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:40385)
0925 05:06:22.546387 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:22.546399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:22.546493 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:22.546653 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:22.546654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:22.546836 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:22.546876 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:22.546877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.547525 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.547526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:22.548075 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:22.548076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.548434 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.548435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:22.548508 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:22.549052 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:22.549057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:22.550705 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
W20260925 05:06:22.550901 23645 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:22.546307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:40385 (local address 127.14.33.62:42263)
0925 05:06:22.546388 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:22.546390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:22.546474 (+ 84us) server_negotiation.cc:407] Connection header received
0925 05:06:22.546549 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:22.546549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:22.546564 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:22.546635 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:22.546951 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.547442 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.548168 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:22.548351 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:22.548432 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:22.550805 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:40385: BlockingRecv error: recv got EOF from 127.14.33.3:40385 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 05:06:23 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:23 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:23.318140 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.313891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:58373)
0925 05:06:23.314015 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.314028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:23.314089 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:23.314207 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:23.314208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:23.314361 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:23.314425 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.314427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.315037 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.315038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.315573 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.315575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.315885 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.315885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.315919 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.316326 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:23.316330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:23.318061 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260925 05:06:23.318228 23649 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:23.313959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:58373 (local address 127.14.33.62:42263)
0925 05:06:23.314063 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:23.314064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:23.314075 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:06:23.314136 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:23.314137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:23.314147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:23.314189 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:23.314488 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.314973 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.315640 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.315831 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.315874 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.318142 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:58373: BlockingRecv error: recv got EOF from 127.14.33.1:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
Sep 25 05:06:23 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:23 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:23.428987 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.424550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:44293)
0925 05:06:23.424679 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.424696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:23.424740 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:23.424880 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:23.424880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:23.425033 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:23.425077 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.425079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.425856 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.425857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.426405 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.426406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.426754 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.426755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.426832 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.427346 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:23.427350 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:23.428919 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
W20260925 05:06:23.429087 23649 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:23.424670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:44293 (local address 127.14.33.62:42263)
0925 05:06:23.424752 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:23.424754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:23.424787 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:06:23.424809 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:23.424810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:23.424820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:23.424861 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:23.425222 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.425795 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.426478 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.426678 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.426758 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.428998 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:44293: BlockingRecv error: recv got EOF from 127.14.33.2:44293 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 05:06:23 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:23 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:23.555981 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:23.551186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:50423)
0925 05:06:23.551334 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:23.551345 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:23.551424 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:23.551564 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:23.551565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:23.551762 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:23.551813 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.551814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.552462 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.552463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.553018 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:23.553020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.553378 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.553379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:23.553432 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.553833 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:23.553837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:23.555874 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
W20260925 05:06:23.556030 23649 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:23.551272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:50423 (local address 127.14.33.62:42263)
0925 05:06:23.551338 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:23.551340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:23.551406 (+ 66us) server_negotiation.cc:407] Connection header received
0925 05:06:23.551487 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:23.551488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:23.551501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:23.551545 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:23.551887 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.552394 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.553103 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:23.553317 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:23.553380 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:23.555961 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:50423: BlockingRecv error: recv got EOF from 127.14.33.3:50423 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 05:06:24 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:24 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:24.322682 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.318610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:45137)
0925 05:06:24.318730 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.318743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.318789 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.318902 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.318902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.319053 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.319099 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.319100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.319692 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.319692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.320154 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.320155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.320446 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.320447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.320482 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.320861 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.320864 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.322622 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
W20260925 05:06:24.322829 23653 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.318667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:45137 (local address 127.14.33.62:42263)
0925 05:06:24.318799 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.318801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.318808 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:24.318831 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.318831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.318845 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.318899 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.319153 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.319630 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.320217 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.320389 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.320433 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.322713 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:45137: BlockingRecv error: recv got EOF from 127.14.33.1:45137 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
Sep 25 05:06:24 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:24 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:24.433714 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.429353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:52329)
0925 05:06:24.429503 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.429514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.429595 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.429727 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.429728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.429906 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.429946 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.429948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.430553 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.430554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.431077 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.431079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.431449 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.431450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.431511 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.431975 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.431980 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.433648 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
W20260925 05:06:24.433785 23653 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.429438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:52329 (local address 127.14.33.62:42263)
0925 05:06:24.429502 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.429503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.429579 (+ 76us) server_negotiation.cc:407] Connection header received
0925 05:06:24.429646 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.429647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.429660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.429719 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.430020 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.430469 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.431159 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.431354 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.431448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.433742 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:52329: BlockingRecv error: recv got EOF from 127.14.33.2:52329 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 05:06:24 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:24 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.3@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:24.560667 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:24.556412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.3:33267)
0925 05:06:24.556553 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:24.556564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:24.556625 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:24.556749 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:24.556750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:24.556905 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:24.556949 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.556951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.557560 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.557561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.558071 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:24.558073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.558365 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.558366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:24.558403 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.558789 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:24.558793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:24.560538 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260925 05:06:24.560738 23653 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:24.556490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:33267 (local address 127.14.33.62:42263)
0925 05:06:24.556557 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:24.556560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:24.556627 (+ 67us) server_negotiation.cc:407] Connection header received
0925 05:06:24.556667 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:24.556668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:24.556681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:24.556731 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:24.557011 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.557500 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.558134 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:24.558307 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:24.558353 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:24.560650 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.3:33267: BlockingRecv error: recv got EOF from 127.14.33.3:33267 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 05:06:25 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:25 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.1@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:25.331552 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.323228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.1:48879)
0925 05:06:25.323388 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:25.323399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:25.323465 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:25.323590 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:25.323590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:25.323753 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:25.323803 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.323805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.324384 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.324385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.324899 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.324901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.325412 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.325413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.325475 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.329779 (+ 4304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:25.329789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:25.331484 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
W20260925 05:06:25.331645 23657 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.323301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:48879 (local address 127.14.33.62:42263)
0925 05:06:25.323416 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.323417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.323465 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:06:25.323517 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.323517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.323526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.323583 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.323866 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.324318 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.325052 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.325352 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.325415 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.331572 (+ 6157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.1:48879: BlockingRecv error: recv got EOF from 127.14.33.1:48879 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
Sep 25 05:06:25 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 05:06:25 dist-test-slave-4dmq krb5kdc[22598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.33.2@KRBTEST.COM for oryx/127.14.33.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 05:06:25.438555 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.434206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:42263 (local address 127.14.33.2:41319)
0925 05:06:25.434374 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:25.434385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:25.434466 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:25.434611 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:25.434612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:25.434807 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:25.434852 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.434853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.435515 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.435516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.436061 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.436063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.436401 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.436401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.436455 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.436902 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:25.436907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:25.438474 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.33.62:42263: Server oryx/127.14.33.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
W20260925 05:06:25.438619 23657 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:25.434284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:41319 (local address 127.14.33.62:42263)
0925 05:06:25.434369 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.434371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.434450 (+ 79us) server_negotiation.cc:407] Connection header received
0925 05:06:25.434521 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.434522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.434535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.434603 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.434937 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.435435 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.436140 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.436334 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.436404 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.438546 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.33.2:41319: BlockingRecv error: recv got EOF from 127.14.33.2:41319 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:06:25.484930 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23182
I20260925 05:06:25.489244 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23333
I20260925 05:06:25.493201 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23466
I20260925 05:06:25.497346 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23111
2026-09-25T05:06:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17276 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T05:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:25Z Disabled control of system clock
I20260925 05:06:25.522964 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:33951
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:33951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 05:06:25.598690 23662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:25.598865 23662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:25.598883 23662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:25.599943 23662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:25.599979 23662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:25.599992 23662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:25.600024 23662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:25.600052 23662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:25.601260 23662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:33951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:33951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:25.601481 23662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:25.601743 23662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:25.606294 23667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:25.606314 23670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:25.606611 23662 server_base.cc:1034] running on GCE node
W20260925 05:06:25.606630 23668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:25.606892 23662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:25.607172 23662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:25.608331 23662 hybrid_clock.cc:648] HybridClock initialized: now 1790312785608308 us; error 34 us; skew 500 ppm
I20260925 05:06:25.609531 23662 webserver.cc:460] Webserver started at http://127.14.33.62:37289/ using document root <none> and password file <none>
I20260925 05:06:25.609771 23662 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:25.609838 23662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:25.609977 23662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:25.610932 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "6279fe904afc4e5b9f18d96661f2bc49"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "8e2a305c6e7fad07d9c5ae285544f31e"
server_key_iv: "08a0ea569e904413b03005cc9938900b"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.611281 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "6279fe904afc4e5b9f18d96661f2bc49"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "8e2a305c6e7fad07d9c5ae285544f31e"
server_key_iv: "08a0ea569e904413b03005cc9938900b"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.612471 23662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:06:25.613188 23676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.613373 23662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:25.613466 23662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "6279fe904afc4e5b9f18d96661f2bc49"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "8e2a305c6e7fad07d9c5ae285544f31e"
server_key_iv: "08a0ea569e904413b03005cc9938900b"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.613546 23662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:25.631682 23662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:25.631955 23662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:25.632087 23662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:25.635690 23662 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:33951
I20260925 05:06:25.635720 23728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:33951 every 8 connection(s)
I20260925 05:06:25.636034 23662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:25.636576 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23662
I20260925 05:06:25.636677 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:06:25.636758 23729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:25.636874 14468 external_mini_cluster.cc:1428] Setting key a4001a764455872df3ef84027f6ed934
I20260925 05:06:25.640121 23729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49: Bootstrap starting.
I20260925 05:06:25.640719 23729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:25.641043 23729 log.cc:824] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:25.641839 23729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49: No bootstrap required, opened a new log
I20260925 05:06:25.642304 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.637359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.14.33.62:33951)
0925 05:06:25.637538 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.637548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.637576 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:06:25.638178 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.638184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.638241 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.638346 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.638568 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.639391 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.640474 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.640906 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.640989 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.641639 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:25.641643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:25.641643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:25.641835 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:25.641874 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:25.641888 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:25.641957 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:25.642036 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 05:06:25.642104 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260925 05:06:25.643551 23729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6279fe904afc4e5b9f18d96661f2bc49" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33951 } }
I20260925 05:06:25.643661 23729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:25.643682 23729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6279fe904afc4e5b9f18d96661f2bc49, State: Initialized, Role: FOLLOWER
I20260925 05:06:25.643810 23729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6279fe904afc4e5b9f18d96661f2bc49" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33951 } }
I20260925 05:06:25.643910 23729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:25.643954 23729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:25.643999 23729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:25.644485 23729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6279fe904afc4e5b9f18d96661f2bc49" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33951 } }
I20260925 05:06:25.644594 23729 leader_election.cc:302] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6279fe904afc4e5b9f18d96661f2bc49; no voters:
I20260925 05:06:25.644798 23729 leader_election.cc:288] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:25.644865 23734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:25.645031 23734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [term 1 LEADER]: Becoming Leader. State: Replica: 6279fe904afc4e5b9f18d96661f2bc49, State: Running, Role: LEADER
I20260925 05:06:25.645062 23729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:25.645140 23734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6279fe904afc4e5b9f18d96661f2bc49" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33951 } }
I20260925 05:06:25.645493 23735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6279fe904afc4e5b9f18d96661f2bc49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6279fe904afc4e5b9f18d96661f2bc49" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33951 } } }
I20260925 05:06:25.645536 23736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6279fe904afc4e5b9f18d96661f2bc49. Latest consensus state: current_term: 1 leader_uuid: "6279fe904afc4e5b9f18d96661f2bc49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6279fe904afc4e5b9f18d96661f2bc49" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33951 } } }
I20260925 05:06:25.645686 23736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:25.645880 23743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:25.646059 23735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49 [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:25.646338 23743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:25.647755 23743 catalog_manager.cc:1347] Generated new cluster ID: bca4e27bab3345c99f043370b2619c94
I20260925 05:06:25.647801 23743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:25.659639 23743 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:25.660235 23743 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:25.664321 23743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6279fe904afc4e5b9f18d96661f2bc49: Generated new TSK 0
I20260925 05:06:25.664469 23743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:25.667229 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33951
--builtin_ntp_servers=127.14.33.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:06:25.747020 23753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:25.747193 23753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:25.747211 23753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:25.748286 23753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:25.748322 23753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:25.748344 23753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:25.749610 23753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:25.749811 23753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:25.750094 23753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:25.754707 23759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:25.754736 23758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:25.754678 23761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:25.755015 23753 server_base.cc:1034] running on GCE node
I20260925 05:06:25.755213 23753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:25.755452 23753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:25.756626 23753 hybrid_clock.cc:648] HybridClock initialized: now 1790312785756617 us; error 29 us; skew 500 ppm
I20260925 05:06:25.757958 23753 webserver.cc:460] Webserver started at http://127.14.33.1:45977/ using document root <none> and password file <none>
I20260925 05:06:25.758181 23753 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:25.758237 23753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:25.758374 23753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:25.759690 23753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "44e1a4d688b4428ba5870345e48d3e35"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "2f85880a080fe13777b7dc48a3a36c13"
server_key_iv: "2d98ca3ef8cf567e6f9b3212e0030fbf"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.760102 23753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "44e1a4d688b4428ba5870345e48d3e35"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "2f85880a080fe13777b7dc48a3a36c13"
server_key_iv: "2d98ca3ef8cf567e6f9b3212e0030fbf"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.761763 23753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 05:06:25.762562 23767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.762774 23753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.762859 23753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "44e1a4d688b4428ba5870345e48d3e35"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "2f85880a080fe13777b7dc48a3a36c13"
server_key_iv: "2d98ca3ef8cf567e6f9b3212e0030fbf"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.762926 23753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:25.782068 23753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:25.782305 23753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:25.782409 23753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:25.782586 23753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:25.782848 23753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:25.782879 23753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.782902 23753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:25.782922 23753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.788931 23753 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:35055
I20260925 05:06:25.788962 23880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:35055 every 8 connection(s)
I20260925 05:06:25.789317 23753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:25.791991 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23753
I20260925 05:06:25.792096 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:25.792282 14468 external_mini_cluster.cc:1428] Setting key 05afa2202225cb1d5d9df66289894639
I20260925 05:06:25.793293 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33951
--builtin_ntp_servers=127.14.33.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 05:06:25.794638 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.790001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:46881 (local address 127.14.33.62:33951)
0925 05:06:25.790045 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.790047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.790291 (+ 244us) server_negotiation.cc:407] Connection header received
0925 05:06:25.790760 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.790761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.790773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.790822 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.791382 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.791859 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.793202 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.793433 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.793492 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.794347 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:25.794348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:25.794349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:25.794352 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:25.794366 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:25.794369 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:25.794406 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:25.794486 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:06:25.794516 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:06:25.794848 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.790006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33951 (local address 127.14.33.1:46881)
0925 05:06:25.790178 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:25.790209 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:25.790673 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:25.790869 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:25.790872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:25.790879 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:25.791315 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.791318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.791958 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.791960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.793085 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.793088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.793505 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.793506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.793681 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.794188 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:06:25.794189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:25.794225 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:06:25.794252 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:25.794409 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:25.794412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:25.794415 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:25.794491 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 05:06:25.794602 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260925 05:06:25.795351 23881 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33951
I20260925 05:06:25.795451 23881 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:25.795631 23881 heartbeater.cc:511] Master 127.14.33.62:33951 requested a full tablet report, sending...
I20260925 05:06:25.796075 23693 ts_manager.cc:194] Registered new tserver with Master: 44e1a4d688b4428ba5870345e48d3e35 (127.14.33.1:35055)
I20260925 05:06:25.796661 23693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.1:46881
W20260925 05:06:25.868593 23884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:25.868760 23884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:25.868779 23884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:25.869870 23884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:25.869906 23884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:25.869930 23884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:25.871199 23884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:25.871412 23884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:25.871695 23884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:25.876443 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:25.876492 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:25.876487 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:25.876636 23884 server_base.cc:1034] running on GCE node
I20260925 05:06:25.876842 23884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:25.877101 23884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:25.878295 23884 hybrid_clock.cc:648] HybridClock initialized: now 1790312785878278 us; error 31 us; skew 500 ppm
I20260925 05:06:25.879415 23884 webserver.cc:460] Webserver started at http://127.14.33.2:35021/ using document root <none> and password file <none>
I20260925 05:06:25.879585 23884 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:25.879623 23884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:25.879714 23884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:25.880630 23884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "17cb8f481fae4fe28c9b881b2e822477"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "530f8f7f8f00280ce9b18f105de3d7d3"
server_key_iv: "f031c6b1a8e2a36a64b54d7cd3be3286"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.880905 23884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "17cb8f481fae4fe28c9b881b2e822477"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "530f8f7f8f00280ce9b18f105de3d7d3"
server_key_iv: "f031c6b1a8e2a36a64b54d7cd3be3286"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.882133 23884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 05:06:25.882865 23898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.883037 23884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:06:25.883103 23884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "17cb8f481fae4fe28c9b881b2e822477"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "530f8f7f8f00280ce9b18f105de3d7d3"
server_key_iv: "f031c6b1a8e2a36a64b54d7cd3be3286"
server_key_version: "encryptionkey@0"
I20260925 05:06:25.883152 23884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:25.903012 23884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:25.903201 23884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:25.903285 23884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:25.903421 23884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:25.903630 23884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:25.903656 23884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.903674 23884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:25.903687 23884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:25.908779 23884 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:42971
I20260925 05:06:25.908854 24011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:42971 every 8 connection(s)
I20260925 05:06:25.909149 23884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:25.913991 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.909810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:37177 (local address 127.14.33.62:33951)
0925 05:06:25.909869 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:25.909871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:25.910203 (+ 332us) server_negotiation.cc:407] Connection header received
0925 05:06:25.910682 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:25.910683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:25.910695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:25.910768 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:25.911379 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.911833 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.912694 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.912892 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.912992 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.913700 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:25.913710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:25.913710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:25.913715 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:25.913730 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:25.913734 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:25.913785 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:25.913879 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 05:06:25.913903 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:06:25.914172 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:25.909808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33951 (local address 127.14.33.2:37177)
0925 05:06:25.909984 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:25.910011 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:25.910577 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:25.910831 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:25.910834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:25.910841 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:25.911285 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.911289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.911918 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.911919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.912597 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:25.912599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:25.913002 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:25.913004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:25.913157 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:25.913573 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:06:25.913573 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:25.913597 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:06:25.913617 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:25.913795 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:25.913800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:25.913803 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:25.913881 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:06:25.913981 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260925 05:06:25.914619 24012 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33951
I20260925 05:06:25.914685 24012 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:25.914867 24012 heartbeater.cc:511] Master 127.14.33.62:33951 requested a full tablet report, sending...
I20260925 05:06:25.915216 23693 ts_manager.cc:194] Registered new tserver with Master: 17cb8f481fae4fe28c9b881b2e822477 (127.14.33.2:42971)
I20260925 05:06:25.915589 23693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.2:37177
I20260925 05:06:25.918490 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 23884
I20260925 05:06:25.918597 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:25.918766 14468 external_mini_cluster.cc:1428] Setting key 7925a555a52a0226c39ba53a77c9fdf9
I20260925 05:06:25.919525 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33951
--builtin_ntp_servers=127.14.33.20:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 05:06:25.992141 24015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:25.992303 24015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:25.992321 24015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:25.993342 24015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:25.993373 24015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:25.993395 24015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:25.994506 24015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:25.994675 24015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:25.994905 24015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:25.999292 24020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:25.999336 24015 server_base.cc:1034] running on GCE node
W20260925 05:06:25.999531 24023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:25.999336 24021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:25.999837 24015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:26.000048 24015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:26.001174 24015 hybrid_clock.cc:648] HybridClock initialized: now 1790312786001158 us; error 44 us; skew 500 ppm
I20260925 05:06:26.002313 24015 webserver.cc:460] Webserver started at http://127.14.33.3:46131/ using document root <none> and password file <none>
I20260925 05:06:26.002542 24015 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:26.002588 24015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:26.002740 24015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:26.003768 24015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "177bdcb08d6d47f58bb4a2229ee1b05e"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "acc3e1c0733356583beb8144d07c6767"
server_key_iv: "167aa4f92f1e389635c4c96bda0e2849"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.004145 24015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "177bdcb08d6d47f58bb4a2229ee1b05e"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "acc3e1c0733356583beb8144d07c6767"
server_key_iv: "167aa4f92f1e389635c4c96bda0e2849"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.005367 24015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:06:26.006034 24029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.006301 24015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:06:26.006412 24015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "177bdcb08d6d47f58bb4a2229ee1b05e"
format_stamp: "Formatted at 2026-09-25 05:06:25 on dist-test-slave-4dmq"
server_key: "acc3e1c0733356583beb8144d07c6767"
server_key_iv: "167aa4f92f1e389635c4c96bda0e2849"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.006497 24015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:26.033915 24015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:26.034235 24015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:26.034380 24015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:26.034590 24015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:26.034897 24015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:26.034960 24015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.034999 24015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:26.035025 24015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.040669 24015 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:40793
I20260925 05:06:26.040709 24142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:40793 every 8 connection(s)
I20260925 05:06:26.041044 24015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:26.043927 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24015
I20260925 05:06:26.044021 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:26.044191 14468 external_mini_cluster.cc:1428] Setting key 86e9cbea59197c7211c1ab6efa564d4d
I20260925 05:06:26.046078 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.041768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:44649 (local address 127.14.33.62:33951)
0925 05:06:26.041845 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.041848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.042036 (+ 188us) server_negotiation.cc:407] Connection header received
0925 05:06:26.042453 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.042454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.042464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.042509 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.043073 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.043532 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.044817 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.045078 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.045160 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.045867 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:26.045868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:26.045869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:26.045872 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:26.045884 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:26.045887 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:26.045944 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.046007 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.046024 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:06:26.046310 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.041745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33951 (local address 127.14.33.3:44649)
0925 05:06:26.041929 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:26.041958 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:26.042379 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:26.042562 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:26.042565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:26.042572 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:26.043013 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:26.043017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.043606 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.043607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:26.044753 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:26.044755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:26.045167 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:26.045168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:26.045315 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:26.045769 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:06:26.045769 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:26.045793 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:06:26.045812 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:26.045960 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:26.045962 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:26.045964 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:06:26.046007 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 05:06:26.046073 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260925 05:06:26.046823 24143 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33951
I20260925 05:06:26.046890 24143 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:26.047077 24143 heartbeater.cc:511] Master 127.14.33.62:33951 requested a full tablet report, sending...
I20260925 05:06:26.047417 23693 ts_manager.cc:194] Registered new tserver with Master: 177bdcb08d6d47f58bb4a2229ee1b05e (127.14.33.3:40793)
I20260925 05:06:26.047776 23693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.3:44649
I20260925 05:06:26.055761 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:06:26.057454 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:26.056995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.14.33.62:33951)
0925 05:06:26.057067 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.057069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.057213 (+ 144us) server_negotiation.cc:407] Connection header received
0925 05:06:26.057256 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.057256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.057264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.057299 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.057398 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41690: BlockingRecv error: recv got EOF from 127.0.0.1:41690 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:06:26.057551 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23753
I20260925 05:06:26.061579 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23884
I20260925 05:06:26.065546 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24015
I20260925 05:06:26.069365 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 23662
2026-09-25T05:06:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (571 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T05:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:26Z Disabled control of system clock
I20260925 05:06:26.093731 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:33631
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:33631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:06:26.164667 24154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:26.164878 24154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:26.164909 24154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:26.166064 24154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:26.166126 24154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:26.166145 24154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:26.166167 24154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:26.166196 24154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:26.167404 24154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:33631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:33631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:26.167637 24154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:26.167881 24154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:26.172312 24162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.172324 24159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.172322 24160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:26.172451 24154 server_base.cc:1034] running on GCE node
I20260925 05:06:26.172636 24154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:26.172870 24154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:26.174009 24154 hybrid_clock.cc:648] HybridClock initialized: now 1790312786174001 us; error 26 us; skew 500 ppm
I20260925 05:06:26.175069 24154 webserver.cc:460] Webserver started at http://127.14.33.62:44763/ using document root <none> and password file <none>
I20260925 05:06:26.175288 24154 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:26.175351 24154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:26.175495 24154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:26.176411 24154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "627785f359144637831958ebb15724fd"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "e4fbb36bfb4cf8f87ec6bb951276929f"
server_key_iv: "73cd270191fe74eabb8b253497ad00da"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.176743 24154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "627785f359144637831958ebb15724fd"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "e4fbb36bfb4cf8f87ec6bb951276929f"
server_key_iv: "73cd270191fe74eabb8b253497ad00da"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.178025 24154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:06:26.178805 24168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.178987 24154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:26.179078 24154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "627785f359144637831958ebb15724fd"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "e4fbb36bfb4cf8f87ec6bb951276929f"
server_key_iv: "73cd270191fe74eabb8b253497ad00da"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.179157 24154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:26.182302 24154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:26.182536 24154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:26.182686 24154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:26.186841 24154 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:33631
I20260925 05:06:26.187196 24220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:33631 every 8 connection(s)
I20260925 05:06:26.187228 24154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:26.189090 24221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:26.192673 24221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd: Bootstrap starting.
I20260925 05:06:26.193373 24221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:26.193714 24221 log.cc:824] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:26.194320 24221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd: No bootstrap required, opened a new log
I20260925 05:06:26.195813 24221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627785f359144637831958ebb15724fd" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33631 } }
I20260925 05:06:26.195955 24221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:26.195998 24221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627785f359144637831958ebb15724fd, State: Initialized, Role: FOLLOWER
I20260925 05:06:26.196130 24221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627785f359144637831958ebb15724fd" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33631 } }
I20260925 05:06:26.196211 24221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:26.196254 24221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:26.196300 24221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:26.196797 24221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627785f359144637831958ebb15724fd" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33631 } }
I20260925 05:06:26.196902 24221 leader_election.cc:302] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627785f359144637831958ebb15724fd; no voters:
I20260925 05:06:26.197088 24221 leader_election.cc:288] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:26.197157 24224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:26.197304 24224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [term 1 LEADER]: Becoming Leader. State: Replica: 627785f359144637831958ebb15724fd, State: Running, Role: LEADER
I20260925 05:06:26.197297 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24154
I20260925 05:06:26.197372 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:06:26.197368 24221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:26.197369 24224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627785f359144637831958ebb15724fd" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33631 } }
I20260925 05:06:26.197568 14468 external_mini_cluster.cc:1428] Setting key ced19941d166d2d254ec91bf385cb8b5
I20260925 05:06:26.197662 24225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627785f359144637831958ebb15724fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627785f359144637831958ebb15724fd" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33631 } } }
I20260925 05:06:26.197654 24224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627785f359144637831958ebb15724fd. Latest consensus state: current_term: 1 leader_uuid: "627785f359144637831958ebb15724fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627785f359144637831958ebb15724fd" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33631 } } }
I20260925 05:06:26.197755 24225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:26.197757 24224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd [sys.catalog]: This master's current role is: LEADER
W20260925 05:06:26.199066 24242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:26.199131 24242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:06:26.199200 24230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:26.199404 24230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:26.200219 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.198345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.14.33.62:33631)
0925 05:06:26.198535 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.198543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.198781 (+ 238us) server_negotiation.cc:407] Connection header received
0925 05:06:26.198976 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.198983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.199027 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.199402 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.199810 (+ 408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:26.199814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:26.199814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:26.199817 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:26.199852 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:26.199865 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:26.199939 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.200013 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.200065 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260925 05:06:26.200963 24230 catalog_manager.cc:1347] Generated new cluster ID: e00ee673db4549259d65c5eaeff0e950
I20260925 05:06:26.201033 24230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:26.209929 24230 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:26.210067 24230 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:26.223287 24230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627785f359144637831958ebb15724fd: Generated new TSK 0
I20260925 05:06:26.223446 24230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:06:26.229691 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33631
--builtin_ntp_servers=127.14.33.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:06:26.310662 24245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:26.310858 24245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:26.310880 24245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:26.311969 24245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:26.312003 24245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:26.312026 24245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:26.313426 24245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:26.313639 24245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:26.313972 24245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:26.318662 24251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.318753 24253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.318665 24250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:26.318871 24245 server_base.cc:1034] running on GCE node
I20260925 05:06:26.319150 24245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:26.319406 24245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:26.320554 24245 hybrid_clock.cc:648] HybridClock initialized: now 1790312786320546 us; error 26 us; skew 500 ppm
I20260925 05:06:26.327145 24245 webserver.cc:460] Webserver started at http://127.14.33.1:33657/ using document root <none> and password file <none>
I20260925 05:06:26.327371 24245 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:26.327414 24245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:26.327538 24245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:26.328502 24245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "b1125077f6044ebd852c9ea43738073c"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "6b3304938241ed66146365387e659880"
server_key_iv: "e1013f5d1cbb78f35a6bea5abcd24ba1"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.328835 24245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "b1125077f6044ebd852c9ea43738073c"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "6b3304938241ed66146365387e659880"
server_key_iv: "e1013f5d1cbb78f35a6bea5abcd24ba1"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.330092 24245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:06:26.330811 24259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.330950 24245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:26.331027 24245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "b1125077f6044ebd852c9ea43738073c"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "6b3304938241ed66146365387e659880"
server_key_iv: "e1013f5d1cbb78f35a6bea5abcd24ba1"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.331087 24245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:26.334221 24245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:26.334478 24245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:26.334594 24245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:26.334808 24245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:26.335088 24245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:26.335117 24245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.335152 24245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:26.335176 24245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.340530 24245 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:36159
I20260925 05:06:26.340601 24372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:36159 every 8 connection(s)
I20260925 05:06:26.340870 24245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:26.343521 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.341563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:46877 (local address 127.14.33.62:33631)
0925 05:06:26.341609 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.341611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.341796 (+ 185us) server_negotiation.cc:407] Connection header received
0925 05:06:26.342190 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.342191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.342197 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.342256 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.343313 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:26.343315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:26.343315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:26.343319 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:26.343334 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:26.343338 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:26.343391 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.343465 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.343476 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:06:26.343685 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.341523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33631 (local address 127.14.33.1:46877)
0925 05:06:26.341696 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:26.341730 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:26.342106 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:26.342312 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:26.342315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:26.342322 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:26.343081 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:06:26.343081 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:26.343104 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:06:26.343126 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:26.343401 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:26.343406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:26.343407 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 05:06:26.343473 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 05:06:26.343513 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260925 05:06:26.344209 24373 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33631
I20260925 05:06:26.344304 24373 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:26.344425 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24245
I20260925 05:06:26.344444 24373 heartbeater.cc:511] Master 127.14.33.62:33631 requested a full tablet report, sending...
I20260925 05:06:26.344508 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:26.344662 14468 external_mini_cluster.cc:1428] Setting key 41192eb9a86bc74c3e494f12544fb2aa
I20260925 05:06:26.344883 24185 ts_manager.cc:194] Registered new tserver with Master: b1125077f6044ebd852c9ea43738073c (127.14.33.1:36159)
I20260925 05:06:26.345669 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33631
--builtin_ntp_servers=127.14.33.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:06:26.418956 24376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:26.419152 24376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:26.419194 24376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:26.420225 24376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:26.420279 24376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:26.420316 24376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:26.421522 24376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:26.421801 24376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:26.422086 24376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:26.426522 24381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.426515 24382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.426530 24384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:26.426734 24376 server_base.cc:1034] running on GCE node
I20260925 05:06:26.426942 24376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:26.427160 24376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:26.428339 24376 hybrid_clock.cc:648] HybridClock initialized: now 1790312786428324 us; error 34 us; skew 500 ppm
I20260925 05:06:26.430012 24376 webserver.cc:460] Webserver started at http://127.14.33.2:39853/ using document root <none> and password file <none>
I20260925 05:06:26.430296 24376 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:26.430353 24376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:26.430488 24376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:26.431826 24376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "f52e09e10d0d478194fc3e05b8672434"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "650f93abc6116b2fdf11be2ee1689d6b"
server_key_iv: "54bbac93a28bd0e4a4975c25e81de550"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.432211 24376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "f52e09e10d0d478194fc3e05b8672434"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "650f93abc6116b2fdf11be2ee1689d6b"
server_key_iv: "54bbac93a28bd0e4a4975c25e81de550"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.433954 24376 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 05:06:26.435009 24390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.435215 24376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:06:26.435295 24376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "f52e09e10d0d478194fc3e05b8672434"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "650f93abc6116b2fdf11be2ee1689d6b"
server_key_iv: "54bbac93a28bd0e4a4975c25e81de550"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.435352 24376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:26.439020 24376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:26.439260 24376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:26.439445 24376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:26.439644 24376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:26.439970 24376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:26.440011 24376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.440048 24376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:26.440068 24376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.446501 24376 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:40497
I20260925 05:06:26.446590 24503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:40497 every 8 connection(s)
I20260925 05:06:26.446863 24376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:06:26.449596 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24376
I20260925 05:06:26.449689 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:26.449875 14468 external_mini_cluster.cc:1428] Setting key 4f25b981ec3b4105f53b9404cb42b741
I20260925 05:06:26.450198 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.447463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:52141 (local address 127.14.33.62:33631)
0925 05:06:26.447514 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.447516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.447730 (+ 214us) server_negotiation.cc:407] Connection header received
0925 05:06:26.448368 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.448369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.448376 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.448453 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.449833 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:26.449835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:26.449835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:26.449840 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:26.449859 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:26.449864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:26.449894 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.450108 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.450125 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:06:26.450659 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.447451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33631 (local address 127.14.33.2:52141)
0925 05:06:26.447606 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:26.447660 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:26.448214 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:26.448529 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:26.448533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:26.448541 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:26.449614 (+ 1073us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:06:26.449615 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:26.449657 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:06:26.449716 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:26.449938 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:26.449943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:26.449946 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:26.450112 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 05:06:26.450429 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260925 05:06:26.451138 24504 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33631
I20260925 05:06:26.451264 24504 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:26.451395 24504 heartbeater.cc:511] Master 127.14.33.62:33631 requested a full tablet report, sending...
I20260925 05:06:26.451794 24184 ts_manager.cc:194] Registered new tserver with Master: f52e09e10d0d478194fc3e05b8672434 (127.14.33.2:40497)
I20260925 05:06:26.453281 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33631
--builtin_ntp_servers=127.14.33.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 05:06:26.530565 24507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:26.530726 24507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:26.530753 24507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:26.531773 24507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:26.531803 24507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:26.531826 24507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:26.532972 24507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:26.533167 24507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:26.533391 24507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:26.537709 24512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:26.537760 24507 server_base.cc:1034] running on GCE node
W20260925 05:06:26.537715 24513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:26.537710 24515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:26.538089 24507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:26.538303 24507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:26.539469 24507 hybrid_clock.cc:648] HybridClock initialized: now 1790312786539452 us; error 32 us; skew 500 ppm
I20260925 05:06:26.540635 24507 webserver.cc:460] Webserver started at http://127.14.33.3:40985/ using document root <none> and password file <none>
I20260925 05:06:26.540858 24507 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:26.540922 24507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:26.541057 24507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:26.542076 24507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "2339040a32274ddeb40c9686e1ea1b54"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "7f6219db40d6812a68b9aa390f2e77d3"
server_key_iv: "8beec82d3775d3ab85cf34a112402b50"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.542439 24507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "2339040a32274ddeb40c9686e1ea1b54"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "7f6219db40d6812a68b9aa390f2e77d3"
server_key_iv: "8beec82d3775d3ab85cf34a112402b50"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.543627 24507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:06:26.544272 24521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.544450 24507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:06:26.544535 24507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "2339040a32274ddeb40c9686e1ea1b54"
format_stamp: "Formatted at 2026-09-25 05:06:26 on dist-test-slave-4dmq"
server_key: "7f6219db40d6812a68b9aa390f2e77d3"
server_key_iv: "8beec82d3775d3ab85cf34a112402b50"
server_key_version: "encryptionkey@0"
I20260925 05:06:26.544613 24507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:26.547518 24507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:26.547758 24507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:26.547885 24507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:26.548079 24507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:26.548348 24507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:26.548400 24507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.548446 24507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:26.548489 24507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:26.553663 24507 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:41701
I20260925 05:06:26.553714 24634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:41701 every 8 connection(s)
I20260925 05:06:26.553963 24507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:26.556344 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.554593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:50363 (local address 127.14.33.62:33631)
0925 05:06:26.554647 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.554649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.554819 (+ 170us) server_negotiation.cc:407] Connection header received
0925 05:06:26.555188 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.555189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.555194 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.555242 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.556191 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:26.556191 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:26.556192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:06:26.556194 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:26.556204 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:06:26.556206 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:26.556242 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:26.556286 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 05:06:26.556297 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:06:26.556507 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:26.554562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33631 (local address 127.14.33.3:50363)
0925 05:06:26.554711 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:26.554751 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:26.555104 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:26.555283 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:26.555286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:26.555292 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:26.556073 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:06:26.556074 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:26.556096 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:06:26.556119 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:26.556243 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:26.556246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:26.556248 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:06:26.556292 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 05:06:26.556330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260925 05:06:26.556928 24635 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33631
I20260925 05:06:26.557013 24635 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:26.557152 24635 heartbeater.cc:511] Master 127.14.33.62:33631 requested a full tablet report, sending...
I20260925 05:06:26.557497 24184 ts_manager.cc:194] Registered new tserver with Master: 2339040a32274ddeb40c9686e1ea1b54 (127.14.33.3:41701)
I20260925 05:06:26.557857 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24507
I20260925 05:06:26.557942 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:26.558094 14468 external_mini_cluster.cc:1428] Setting key 554833f16afcab004293801325045df9
I20260925 05:06:26.558876 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 05:06:26.560565 24236 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:06:26.560100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.14.33.62:33631)
0925 05:06:26.560138 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:26.560139 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:26.560261 (+ 122us) server_negotiation.cc:407] Connection header received
0925 05:06:26.560329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:26.560330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:26.560335 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:26.560377 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:26.560512 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50622: BlockingRecv error: recv got EOF from 127.0.0.1:50622 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:06:26.560626 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24245
I20260925 05:06:26.565301 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24376
I20260925 05:06:26.569940 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24507
I20260925 05:06:26.573527 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24154
2026-09-25T05:06:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (504 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:26 dist-test-slave-4dmq krb5kdc[24645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:26 dist-test-slave-4dmq krb5kdc[24645](info): set up 2 sockets
Sep 25 05:06:26 dist-test-slave-4dmq krb5kdc[24645](info): commencing operation
krb5kdc: starting...
W20260925 05:06:27.538785 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:06:27 dist-test-slave-4dmq krb5kdc[24645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:06:27.647136 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:34889
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:34889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:27.722633 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:27.722819 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:27.722841 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:27.723891 24658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:06:27.723929 24658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:27.723942 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:27.723954 24658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:06:27.723966 24658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:06:27.725358 24658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:34889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:34889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:27.725541 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:27.725695 24658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:27.730413 24664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:27.730472 24663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:27.730489 24666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:27.730659 24658 server_base.cc:1034] running on GCE node
I20260925 05:06:27.730933 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:27.731240 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:27.732409 24658 hybrid_clock.cc:648] HybridClock initialized: now 1790312787732397 us; error 25 us; skew 500 ppm
Sep 25 05:06:27 dist-test-slave-4dmq krb5kdc[24645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312787, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:27.734949 24658 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:06:27.735365 24658 webserver.cc:460] Webserver started at http://127.14.33.62:43901/ using document root <none> and password file <none>
I20260925 05:06:27.735574 24658 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:27.735622 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:27.735772 24658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:27.736734 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "3a04f9413ed3472ebba90e10fcfa224a"
format_stamp: "Formatted at 2026-09-25 05:06:27 on dist-test-slave-4dmq"
server_key: "c7d29172f46f37a0b8e5e8b8f30d4cfa"
server_key_iv: "7f35daae7155784abe164d3887b09996"
server_key_version: "encryptionkey@0"
I20260925 05:06:27.737082 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "3a04f9413ed3472ebba90e10fcfa224a"
format_stamp: "Formatted at 2026-09-25 05:06:27 on dist-test-slave-4dmq"
server_key: "c7d29172f46f37a0b8e5e8b8f30d4cfa"
server_key_iv: "7f35daae7155784abe164d3887b09996"
server_key_version: "encryptionkey@0"
I20260925 05:06:27.738408 24658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:27.739099 24673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:27.739286 24658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:27.739391 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "3a04f9413ed3472ebba90e10fcfa224a"
format_stamp: "Formatted at 2026-09-25 05:06:27 on dist-test-slave-4dmq"
server_key: "c7d29172f46f37a0b8e5e8b8f30d4cfa"
server_key_iv: "7f35daae7155784abe164d3887b09996"
server_key_version: "encryptionkey@0"
I20260925 05:06:27.739463 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:27.760871 24658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:27.763058 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:27.763213 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:27.767060 24658 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:34889
I20260925 05:06:27.767110 24725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:34889 every 8 connection(s)
I20260925 05:06:27.767493 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:06:27.768399 24726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:27.770565 24726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: Bootstrap starting.
I20260925 05:06:27.771116 24726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:27.771322 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24658
I20260925 05:06:27.771423 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:06:27.771391 24726 log.cc:824] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:27.771569 14468 external_mini_cluster.cc:1428] Setting key edf8bb58de451d8a92cfc292d92766d0
I20260925 05:06:27.772212 24726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: No bootstrap required, opened a new log
I20260925 05:06:27.774861 24726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a04f9413ed3472ebba90e10fcfa224a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34889 } }
I20260925 05:06:27.775065 24726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:27.775194 24726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a04f9413ed3472ebba90e10fcfa224a, State: Initialized, Role: FOLLOWER
I20260925 05:06:27.775354 24726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a04f9413ed3472ebba90e10fcfa224a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34889 } }
I20260925 05:06:27.775468 24726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:06:27.775512 24726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:06:27.775559 24726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:27.776087 24726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a04f9413ed3472ebba90e10fcfa224a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34889 } }
I20260925 05:06:27.776232 24726 leader_election.cc:302] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a04f9413ed3472ebba90e10fcfa224a; no voters:
I20260925 05:06:27.776407 24726 leader_election.cc:288] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:06:27.776512 24731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:27.776659 24726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:06:27.777025 24731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [term 1 LEADER]: Becoming Leader. State: Replica: 3a04f9413ed3472ebba90e10fcfa224a, State: Running, Role: LEADER
I20260925 05:06:27.777153 24731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a04f9413ed3472ebba90e10fcfa224a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34889 } }
I20260925 05:06:27.777696 24732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a04f9413ed3472ebba90e10fcfa224a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a04f9413ed3472ebba90e10fcfa224a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34889 } } }
I20260925 05:06:27.777791 24732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [sys.catalog]: This master's current role is: LEADER
I20260925 05:06:27.778129 24733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a04f9413ed3472ebba90e10fcfa224a. Latest consensus state: current_term: 1 leader_uuid: "3a04f9413ed3472ebba90e10fcfa224a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a04f9413ed3472ebba90e10fcfa224a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 34889 } } }
I20260925 05:06:27.778399 24733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a [sys.catalog]: This master's current role is: LEADER
W20260925 05:06:27.778614 24748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:06:27.778657 24748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 05:06:27 dist-test-slave-4dmq krb5kdc[24645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:27.779490 24743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:06:27.779714 24743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:06:27.781533 24743 catalog_manager.cc:1347] Generated new cluster ID: 3f094f4b857a4595883393c4c144c240
I20260925 05:06:27.781581 24743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:06:27.782877 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:27.772076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.14.33.62:34889)
0925 05:06:27.772275 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:27.772284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:27.772319 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:06:27.772770 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:27.772776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:27.772827 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:27.772966 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:27.773602 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:27.774979 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:27.775921 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:27.776360 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:27.776476 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:27.779419 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:27.779423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:27.779424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:27.779436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:27.781291 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:27.781726 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:27.781729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:27.781730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:27.781819 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:27.782047 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:27.782048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:27.782048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:27.782307 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:27.782440 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:27.782563 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 05:06:27.782680 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260925 05:06:27.794996 24743 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:06:27.795686 24743 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:06:27.801345 24743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: Generated new TSK 0
I20260925 05:06:27.801501 24743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:06:27.831547 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34889
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:27.909361 24753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:27.909561 24753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:27.909596 24753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:27.910718 24753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:27.910779 24753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:27.910823 24753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:06:27.912227 24753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:27.912465 24753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:27.912700 24753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:27.917106 24759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:27.917171 24761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:27.917405 24753 server_base.cc:1034] running on GCE node
W20260925 05:06:27.917138 24758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:27.917634 24753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:27.917886 24753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:27.919047 24753 hybrid_clock.cc:648] HybridClock initialized: now 1790312787919041 us; error 33 us; skew 500 ppm
Sep 25 05:06:27 dist-test-slave-4dmq krb5kdc[24645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312787, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:27.921420 24753 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:06:27.921806 24753 webserver.cc:460] Webserver started at http://127.14.33.1:34723/ using document root <none> and password file <none>
I20260925 05:06:27.922026 24753 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:27.922091 24753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:27.922240 24753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:27.923130 24753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "d10e49eea7494a4f97506d4e4bef0103"
format_stamp: "Formatted at 2026-09-25 05:06:27 on dist-test-slave-4dmq"
server_key: "c3be19cc778d4451a8e258ae6edfd692"
server_key_iv: "b2911148f58a31151ec02459ff261a8d"
server_key_version: "encryptionkey@0"
I20260925 05:06:27.923478 24753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "d10e49eea7494a4f97506d4e4bef0103"
format_stamp: "Formatted at 2026-09-25 05:06:27 on dist-test-slave-4dmq"
server_key: "c3be19cc778d4451a8e258ae6edfd692"
server_key_iv: "b2911148f58a31151ec02459ff261a8d"
server_key_version: "encryptionkey@0"
I20260925 05:06:27.924659 24753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 05:06:27.925299 24768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:27.925457 24753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:27.925539 24753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "d10e49eea7494a4f97506d4e4bef0103"
format_stamp: "Formatted at 2026-09-25 05:06:27 on dist-test-slave-4dmq"
server_key: "c3be19cc778d4451a8e258ae6edfd692"
server_key_iv: "b2911148f58a31151ec02459ff261a8d"
server_key_version: "encryptionkey@0"
I20260925 05:06:27.925644 24753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:27.936980 24753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:27.938561 24753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:27.938712 24753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:27.938902 24753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:27.939180 24753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:27.939232 24753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:27.939270 24753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:27.939296 24753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:27.944416 24753 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:41361
I20260925 05:06:27.944478 24881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:41361 every 8 connection(s)
I20260925 05:06:27.944746 24753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:06:27.946027 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24753
I20260925 05:06:27.946130 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:06:27.946302 14468 external_mini_cluster.cc:1428] Setting key e99433e65da76e7b82c8728444f5fcb8
Sep 25 05:06:27 dist-test-slave-4dmq krb5kdc[24645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312787, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
I20260925 05:06:27.953608 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:27.945372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:40399 (local address 127.14.33.62:34889)
0925 05:06:27.945452 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:27.945454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:27.945597 (+ 143us) server_negotiation.cc:407] Connection header received
0925 05:06:27.946035 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:27.946036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:27.946049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:27.946111 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:27.947150 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:27.947625 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:27.948564 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:27.948782 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:27.948864 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:27.950948 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:27.950949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:27.950949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:27.950956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:27.952280 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:27.952674 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:27.952675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:27.952676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:27.952733 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:27.952966 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:27.952967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:27.952967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:27.953107 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:27.953201 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:27.953446 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 05:06:27.953525 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:06:27.953786 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:27.945337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:34889 (local address 127.14.33.1:40399)
0925 05:06:27.945491 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:27.945521 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:27.945923 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:27.946177 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:27.946181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:27.946799 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:27.947048 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:27.947052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:27.947717 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:27.947718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:27.948463 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:27.948465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:27.948875 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:27.948876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:27.949000 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:27.949510 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:27.949517 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:27.950849 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:27.952374 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:27.952381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:27.952383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:27.952584 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:27.952794 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:27.952795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:27.952796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:27.952898 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:27.953194 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:27.953201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:27.953326 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 05:06:27.953426 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:06:27.953560 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260925 05:06:27.954265 24882 heartbeater.cc:347] Connected to a master server at 127.14.33.62:34889
I20260925 05:06:27.954334 24882 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:27.954506 24882 heartbeater.cc:511] Master 127.14.33.62:34889 requested a full tablet report, sending...
I20260925 05:06:27.954984 24690 ts_manager.cc:194] Registered new tserver with Master: d10e49eea7494a4f97506d4e4bef0103 (127.14.33.1:41361)
I20260925 05:06:27.955580 24690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:40399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:06:27.969339 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34889
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:28.042485 24887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:28.042666 24887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:28.042690 24887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:28.043835 24887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:28.043907 24887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:28.043953 24887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:06:28.045392 24887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:28.045608 24887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:28.045822 24887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:28.050486 24892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:28.050511 24893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:28.050515 24895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:28.050738 24887 server_base.cc:1034] running on GCE node
I20260925 05:06:28.050928 24887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:28.051158 24887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:28.052313 24887 hybrid_clock.cc:648] HybridClock initialized: now 1790312788052300 us; error 25 us; skew 500 ppm
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[24645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312788, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:28.054550 24887 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:06:28.054944 24887 webserver.cc:460] Webserver started at http://127.14.33.2:44091/ using document root <none> and password file <none>
I20260925 05:06:28.055135 24887 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:28.055199 24887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:28.055332 24887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:28.056319 24887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "bf9d005af9b148b6a40c35f821ae61cd"
format_stamp: "Formatted at 2026-09-25 05:06:28 on dist-test-slave-4dmq"
server_key: "b1454e61fa296441c4bdb194875c094f"
server_key_iv: "5b5cfbd86d0c815d4c913a8dbd86fe99"
server_key_version: "encryptionkey@0"
I20260925 05:06:28.056656 24887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "bf9d005af9b148b6a40c35f821ae61cd"
format_stamp: "Formatted at 2026-09-25 05:06:28 on dist-test-slave-4dmq"
server_key: "b1454e61fa296441c4bdb194875c094f"
server_key_iv: "5b5cfbd86d0c815d4c913a8dbd86fe99"
server_key_version: "encryptionkey@0"
I20260925 05:06:28.057978 24887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:06:28.058646 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:28.058822 24887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:28.058913 24887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "bf9d005af9b148b6a40c35f821ae61cd"
format_stamp: "Formatted at 2026-09-25 05:06:28 on dist-test-slave-4dmq"
server_key: "b1454e61fa296441c4bdb194875c094f"
server_key_iv: "5b5cfbd86d0c815d4c913a8dbd86fe99"
server_key_version: "encryptionkey@0"
I20260925 05:06:28.058998 24887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:28.066308 24887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:28.068046 24887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:28.068176 24887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:28.068370 24887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:28.068646 24887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:28.068699 24887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:28.068735 24887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:28.068764 24887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:28.074139 24887 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:33489
I20260925 05:06:28.074169 25015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:33489 every 8 connection(s)
I20260925 05:06:28.074473 24887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[24645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312788, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:28.082419 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.075111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:45841 (local address 127.14.33.62:34889)
0925 05:06:28.075168 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.075170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.075320 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:06:28.075699 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.075699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.075708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.075762 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:28.076488 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.077012 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.077929 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.078113 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.078194 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.079937 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:28.079938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:28.079939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:28.079945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:28.081412 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:28.081721 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:28.081722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:28.081722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:28.081771 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:28.081956 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:28.081957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:28.081957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:28.082048 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:28.082094 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.082305 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.082360 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:06:28.082576 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.075081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:34889 (local address 127.14.33.2:45841)
0925 05:06:28.075218 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.075251 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.075629 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.075807 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.075809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.076271 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:28.076426 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.076430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.077097 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.077098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.077838 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.077840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.078246 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.078247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.078335 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.078681 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:28.078688 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:28.079861 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:28.081472 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:28.081477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:28.081478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:28.081660 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:28.081823 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:28.081824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:28.081824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:28.081906 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:28.082097 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:28.082100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:28.082205 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.082278 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.082377 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260925 05:06:28.083037 25016 heartbeater.cc:347] Connected to a master server at 127.14.33.62:34889
I20260925 05:06:28.083146 25016 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:28.083279 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 24887
I20260925 05:06:28.083325 25016 heartbeater.cc:511] Master 127.14.33.62:34889 requested a full tablet report, sending...
I20260925 05:06:28.083382 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:06:28.083570 14468 external_mini_cluster.cc:1428] Setting key 9b6f644bd0034e6bee979bbead762365
I20260925 05:06:28.083709 24690 ts_manager.cc:194] Registered new tserver with Master: bf9d005af9b148b6a40c35f821ae61cd (127.14.33.2:33489)
I20260925 05:06:28.084354 24690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:45841
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:06:28.107309 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34889
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:06:28.181996 25021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:06:28.182168 25021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:06:28.182188 25021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:06:28.183212 25021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:06:28.183247 25021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:06:28.183272 25021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:06:28.184554 25021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:06:28.184716 25021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:06:28.184974 25021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:06:28.191247 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:06:28.191213 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:28.191407 25021 server_base.cc:1034] running on GCE node
W20260925 05:06:28.191533 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:06:28.191779 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:06:28.192028 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:06:28.193192 25021 hybrid_clock.cc:648] HybridClock initialized: now 1790312788193186 us; error 43 us; skew 500 ppm
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[24645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312788, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:06:28.195493 25021 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:06:28.195945 25021 webserver.cc:460] Webserver started at http://127.14.33.3:44259/ using document root <none> and password file <none>
I20260925 05:06:28.196139 25021 fs_manager.cc:359] Metadata directory not provided
I20260925 05:06:28.196177 25021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:06:28.196305 25021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:06:28.197274 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "0862fe55f98545a68ae6013ff7c912b8"
format_stamp: "Formatted at 2026-09-25 05:06:28 on dist-test-slave-4dmq"
server_key: "3714e2a3356ecf9b50eac97cdbdb97b8"
server_key_iv: "1743adc527b48a954d0e90b05cfdd3d1"
server_key_version: "encryptionkey@0"
I20260925 05:06:28.197574 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "0862fe55f98545a68ae6013ff7c912b8"
format_stamp: "Formatted at 2026-09-25 05:06:28 on dist-test-slave-4dmq"
server_key: "3714e2a3356ecf9b50eac97cdbdb97b8"
server_key_iv: "1743adc527b48a954d0e90b05cfdd3d1"
server_key_version: "encryptionkey@0"
I20260925 05:06:28.198710 25021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:06:28.199404 25036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:28.199545 25021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:06:28.199638 25021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "0862fe55f98545a68ae6013ff7c912b8"
format_stamp: "Formatted at 2026-09-25 05:06:28 on dist-test-slave-4dmq"
server_key: "3714e2a3356ecf9b50eac97cdbdb97b8"
server_key_iv: "1743adc527b48a954d0e90b05cfdd3d1"
server_key_version: "encryptionkey@0"
I20260925 05:06:28.199702 25021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:06:28.232651 25021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:06:28.234702 25021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:06:28.234874 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:06:28.235088 25021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:06:28.235375 25021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:06:28.235425 25021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:28.235456 25021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:06:28.235482 25021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:06:28.240747 25021 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:33875
I20260925 05:06:28.240780 25149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:33875 every 8 connection(s)
I20260925 05:06:28.241024 25021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:06:28.241290 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 25021
I20260925 05:06:28.241389 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:06:28.241575 14468 external_mini_cluster.cc:1428] Setting key 1d3ec8891f44e5b17ac0e356f1f1bd92
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[24645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312788, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:06:28.248816 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.241689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:34895 (local address 127.14.33.62:34889)
0925 05:06:28.241747 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.241749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.241936 (+ 187us) server_negotiation.cc:407] Connection header received
0925 05:06:28.242468 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.242469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.242481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.242523 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:28.243347 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.243786 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.244646 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.244812 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.244857 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.246586 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:28.246588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:28.246588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:28.246594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:28.247746 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:28.248104 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:28.248104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:28.248105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:28.248151 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:28.248354 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:28.248355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:28.248355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:28.248450 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:28.248498 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.248701 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.248757 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:06:28.248951 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.241666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:34889 (local address 127.14.33.3:34895)
0925 05:06:28.241836 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.241871 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.242373 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.242589 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.242592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.243122 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:06:28.243282 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.243286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.243867 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.243867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.244584 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.244586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.244877 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.244877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.244960 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.245351 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:06:28.245358 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:06:28.246515 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:06:28.247829 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:28.247834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:28.247835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:28.248043 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:28.248216 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:06:28.248217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:06:28.248217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:06:28.248303 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:06:28.248501 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:06:28.248504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:06:28.248600 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.248673 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.248780 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260925 05:06:28.249394 25150 heartbeater.cc:347] Connected to a master server at 127.14.33.62:34889
I20260925 05:06:28.249487 25150 heartbeater.cc:464] Registering TS with master...
I20260925 05:06:28.249639 25150 heartbeater.cc:511] Master 127.14.33.62:34889 requested a full tablet report, sending...
I20260925 05:06:28.249954 24690 ts_manager.cc:194] Registered new tserver with Master: 0862fe55f98545a68ae6013ff7c912b8 (127.14.33.3:33875)
I20260925 05:06:28.250288 24690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:34895
I20260925 05:06:28.253299 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:06:28.260082 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.254409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.14.33.62:34889)
0925 05:06:28.254465 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.254467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.254538 (+ 71us) server_negotiation.cc:407] Connection header received
0925 05:06:28.254605 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.254605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.254614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.254662 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:06:28.255179 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.255649 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.256396 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.256582 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.256663 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.257800 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:06:28.257801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:06:28.257801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:06:28.257807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:06:28.259023 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:28.259359 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:28.259360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:28.259360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:28.259413 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:06:28.259593 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:06:28.259594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:06:28.259595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:06:28.259726 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:06:28.259825 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.259925 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.259995 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:06:28.261579 24690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:06:28.263123 24690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:06:28.269083 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.265312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:41361 (local address 127.0.0.1:41700)
0925 05:06:28.265436 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.265443 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.265474 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.265715 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.265719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.265721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:28.265803 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.265804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.267028 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.267029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.267819 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.267822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.268765 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.268766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.268844 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.268857 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.268945 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.269001 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260925 05:06:28.269150 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.265426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.14.33.1:41361)
0925 05:06:28.265576 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.265579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.265587 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:28.265615 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.265617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.265627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.265708 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:28.265883 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.266922 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.267908 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.268607 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.268674 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.268729 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.268976 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.269040 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260925 05:06:28.269179 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.265042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:33875 (local address 127.0.0.1:58130)
0925 05:06:28.265175 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.265213 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.265304 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.265964 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.265964 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.265966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:28.266016 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.266017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.266886 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.266887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.267975 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.267977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.268772 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.268773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.268848 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.268858 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.268959 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.269005 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260925 05:06:28.269511 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.265077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:33489 (local address 127.0.0.1:43496)
0925 05:06:28.265214 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.265224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.265290 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.266068 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.266069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.266071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:28.266108 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.266109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.267280 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.267281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.268352 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.268355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.269149 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.269150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.269383 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.269389 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.269417 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.269451 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":104,"threads_started":1}
I20260925 05:06:28.269526 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.265454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.14.33.3:33875)
0925 05:06:28.265810 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.265814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.265822 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:06:28.265851 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.265852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.265864 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.266083 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:28.266139 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.266781 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.268060 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.268694 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.268769 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.268818 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.269418 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.269456 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":67,"threads_started":1}
I20260925 05:06:28.269833 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.265599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.14.33.2:33489)
0925 05:06:28.265751 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.265755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.265790 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:06:28.265832 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.265834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.265849 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.265935 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:28.266186 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.267187 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.268454 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.269043 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.269624 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.269698 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.269722 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.269763 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260925 05:06:28.269804 24816 tablet_service.cc:1467] Processing CreateTablet for tablet 17d31016fbaa4e9e9460838e5ed90894 (DEFAULT_TABLE table=test-table [id=a8349a289c5049fd9c539661b935dd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:28.270045 25084 tablet_service.cc:1467] Processing CreateTablet for tablet 17d31016fbaa4e9e9460838e5ed90894 (DEFAULT_TABLE table=test-table [id=a8349a289c5049fd9c539661b935dd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:28.270151 24816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d31016fbaa4e9e9460838e5ed90894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:28.270401 25084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d31016fbaa4e9e9460838e5ed90894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:28.272498 25164 tablet_bootstrap.cc:492] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: Bootstrap starting.
I20260925 05:06:28.273005 25165 tablet_bootstrap.cc:492] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Bootstrap starting.
I20260925 05:06:28.273280 25164 tablet_bootstrap.cc:654] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:28.273273 24950 tablet_service.cc:1467] Processing CreateTablet for tablet 17d31016fbaa4e9e9460838e5ed90894 (DEFAULT_TABLE table=test-table [id=a8349a289c5049fd9c539661b935dd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:06:28.273604 24950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d31016fbaa4e9e9460838e5ed90894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:06:28.273682 25165 tablet_bootstrap.cc:654] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:28.273854 25164 log.cc:824] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:28.274188 25165 log.cc:824] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:28.274595 25164 tablet_bootstrap.cc:492] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: No bootstrap required, opened a new log
I20260925 05:06:28.274663 25164 ts_tablet_manager.cc:1397] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:06:28.274842 25165 tablet_bootstrap.cc:492] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: No bootstrap required, opened a new log
I20260925 05:06:28.274911 25165 ts_tablet_manager.cc:1397] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:06:28.276070 25165 raft_consensus.cc:348] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.276216 25165 raft_consensus.cc:374] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:28.276288 25165 raft_consensus.cc:729] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0862fe55f98545a68ae6013ff7c912b8, State: Initialized, Role: FOLLOWER
I20260925 05:06:28.276412 25165 consensus_queue.cc:260] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.276556 25164 raft_consensus.cc:348] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.276684 25165 ts_tablet_manager.cc:1428] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:06:28.276705 25164 raft_consensus.cc:374] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:28.276741 25164 raft_consensus.cc:729] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10e49eea7494a4f97506d4e4bef0103, State: Initialized, Role: FOLLOWER
I20260925 05:06:28.276791 25150 heartbeater.cc:503] Master 127.14.33.62:34889 was elected leader, sending a full tablet report...
I20260925 05:06:28.276870 25168 tablet_bootstrap.cc:492] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Bootstrap starting.
I20260925 05:06:28.276861 25164 consensus_queue.cc:260] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.277155 25164 ts_tablet_manager.cc:1428] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:06:28.277261 24882 heartbeater.cc:503] Master 127.14.33.62:34889 was elected leader, sending a full tablet report...
I20260925 05:06:28.277828 25168 tablet_bootstrap.cc:654] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Neither blocks nor log segments found. Creating new log.
I20260925 05:06:28.278394 25168 log.cc:824] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:06:28.279120 25168 tablet_bootstrap.cc:492] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: No bootstrap required, opened a new log
I20260925 05:06:28.279178 25168 ts_tablet_manager.cc:1397] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:06:28.280345 25168 raft_consensus.cc:348] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.280462 25168 raft_consensus.cc:374] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:06:28.280481 25168 raft_consensus.cc:729] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9d005af9b148b6a40c35f821ae61cd, State: Initialized, Role: FOLLOWER
I20260925 05:06:28.280550 25168 consensus_queue.cc:260] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.280786 25016 heartbeater.cc:503] Master 127.14.33.62:34889 was elected leader, sending a full tablet report...
I20260925 05:06:28.281028 25168 ts_tablet_manager.cc:1428] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:06:28.307099 25170 raft_consensus.cc:482] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:06:28.307193 25170 raft_consensus.cc:504] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.307613 25170 leader_election.cc:288] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0862fe55f98545a68ae6013ff7c912b8 (127.14.33.3:33875), bf9d005af9b148b6a40c35f821ae61cd (127.14.33.2:33489)
I20260925 05:06:28.310492 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.307788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:33875 (local address 127.14.33.1:48539)
0925 05:06:28.307846 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.307858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.307902 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.308065 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.308066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.308068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:28.308149 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.308151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.308826 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.308827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.309579 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.309581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.310238 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.310239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.310316 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.310319 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.310372 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.310415 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 05:06:28.310508 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.307852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:43697 (local address 127.14.33.2:33489)
0925 05:06:28.307901 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.307903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.308053 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:06:28.308080 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.308081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.308092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.308148 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:28.308310 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.308758 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.309711 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.310218 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.310274 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.310284 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.310423 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.310457 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:06:28.310503 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.307831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:48539 (local address 127.14.33.3:33875)
0925 05:06:28.307877 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.307879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.307890 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:06:28.307960 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.307961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.307971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.308036 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:06:28.308249 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.308742 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.309662 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.310182 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.310247 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.310256 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.310384 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.310440 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:06:28.310497 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.307790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:33489 (local address 127.14.33.1:43697)
0925 05:06:28.307971 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.307981 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:06:28.308037 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:06:28.308159 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:06:28.308160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:06:28.308161 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:06:28.308240 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.308242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.308853 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.308854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.309605 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:06:28.309607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.310299 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.310300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:06:28.310357 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.310360 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:06:28.310406 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:06:28.310429 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260925 05:06:28.310851 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d31016fbaa4e9e9460838e5ed90894" candidate_uuid: "d10e49eea7494a4f97506d4e4bef0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862fe55f98545a68ae6013ff7c912b8" is_pre_election: true
I20260925 05:06:28.310855 24970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d31016fbaa4e9e9460838e5ed90894" candidate_uuid: "d10e49eea7494a4f97506d4e4bef0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9d005af9b148b6a40c35f821ae61cd" is_pre_election: true
I20260925 05:06:28.311010 24970 raft_consensus.cc:2454] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e49eea7494a4f97506d4e4bef0103 in term 0.
I20260925 05:06:28.311010 25104 raft_consensus.cc:2454] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d10e49eea7494a4f97506d4e4bef0103 in term 0.
I20260925 05:06:28.311251 24771 leader_election.cc:302] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862fe55f98545a68ae6013ff7c912b8, d10e49eea7494a4f97506d4e4bef0103; no voters:
I20260925 05:06:28.311404 25170 raft_consensus.cc:2781] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:06:28.311457 25170 raft_consensus.cc:482] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:06:28.311501 25170 raft_consensus.cc:3037] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:28.312186 25170 raft_consensus.cc:504] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.312318 25170 leader_election.cc:288] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [CANDIDATE]: Term 1 election: Requested vote from peers 0862fe55f98545a68ae6013ff7c912b8 (127.14.33.3:33875), bf9d005af9b148b6a40c35f821ae61cd (127.14.33.2:33489)
I20260925 05:06:28.312491 24970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d31016fbaa4e9e9460838e5ed90894" candidate_uuid: "d10e49eea7494a4f97506d4e4bef0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9d005af9b148b6a40c35f821ae61cd"
I20260925 05:06:28.312511 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d31016fbaa4e9e9460838e5ed90894" candidate_uuid: "d10e49eea7494a4f97506d4e4bef0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862fe55f98545a68ae6013ff7c912b8"
I20260925 05:06:28.312565 25104 raft_consensus.cc:3037] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:28.312565 24970 raft_consensus.cc:3037] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:06:28.313467 24970 raft_consensus.cc:2454] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e49eea7494a4f97506d4e4bef0103 in term 1.
I20260925 05:06:28.313467 25104 raft_consensus.cc:2454] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d10e49eea7494a4f97506d4e4bef0103 in term 1.
I20260925 05:06:28.313628 24769 leader_election.cc:302] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf9d005af9b148b6a40c35f821ae61cd, d10e49eea7494a4f97506d4e4bef0103; no voters:
I20260925 05:06:28.313735 25170 raft_consensus.cc:2781] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:06:28.313874 25170 raft_consensus.cc:686] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [term 1 LEADER]: Becoming Leader. State: Replica: d10e49eea7494a4f97506d4e4bef0103, State: Running, Role: LEADER
I20260925 05:06:28.313982 25170 consensus_queue.cc:237] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } }
I20260925 05:06:28.314749 24690 catalog_manager.cc:5495] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 reported cstate change: term changed from 0 to 1, leader changed from <none> to d10e49eea7494a4f97506d4e4bef0103 (127.14.33.1). New cstate: current_term: 1 leader_uuid: "d10e49eea7494a4f97506d4e4bef0103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d10e49eea7494a4f97506d4e4bef0103" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 41361 } health_report { overall_health: HEALTHY } } }
W20260925 05:06:28.324981 25017 tablet.cc:2369] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:06:28.325759 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.323631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.14.33.1:41361)
0925 05:06:28.323687 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:06:28.323688 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:06:28.323695 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:06:28.323761 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:06:28.323761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:06:28.323775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:06:28.323829 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:06:28.323946 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.324330 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.325075 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:06:28.325302 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:06:28.325351 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:06:28.325487 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:06:28.325565 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:06:28.325618 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:06:28.325653 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 05:06:28.325669 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:06:28.327183 25104 raft_consensus.cc:1261] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 1 FOLLOWER]: Refusing update from remote peer d10e49eea7494a4f97506d4e4bef0103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:28.327193 24970 raft_consensus.cc:1261] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 1 FOLLOWER]: Refusing update from remote peer d10e49eea7494a4f97506d4e4bef0103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:06:28.327471 25174 consensus_queue.cc:1035] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9d005af9b148b6a40c35f821ae61cd" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:28.327606 25170 consensus_queue.cc:1035] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0862fe55f98545a68ae6013ff7c912b8" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:06:28.329586 25177 mvcc.cc:205] Tried to move back new op lower bound from 7333121180986101760 to 7333121180934782976. Current Snapshot: MvccSnapshot[applied={T|T < 7333121180986101760}]
I20260925 05:06:28.331558 24690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:28.331653 24690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:06:28.332332 24690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a04f9413ed3472ebba90e10fcfa224a: Sending DeleteTablet for 3 replicas of tablet 17d31016fbaa4e9e9460838e5ed90894
I20260925 05:06:28.332633 25084 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d31016fbaa4e9e9460838e5ed90894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:28 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:58130
I20260925 05:06:28.332660 24950 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d31016fbaa4e9e9460838e5ed90894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:28 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:43496
I20260925 05:06:28.332875 25187 tablet_replica.cc:330] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: stopping tablet replica
I20260925 05:06:28.332876 24816 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d31016fbaa4e9e9460838e5ed90894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:06:28 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:41700
I20260925 05:06:28.333022 25187 raft_consensus.cc:2229] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:28.333052 25188 tablet_replica.cc:330] T 17d31016fbaa4e9e9460838e5ed90894 P d10e49eea7494a4f97506d4e4bef0103: stopping tablet replica
I20260925 05:06:28.333045 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24753
I20260925 05:06:28.333729 25187 raft_consensus.cc:2258] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:28.333734 25186 tablet_replica.cc:330] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: stopping tablet replica
I20260925 05:06:28.333819 25186 raft_consensus.cc:2229] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:06:28.333909 25186 raft_consensus.cc:2258] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:06:28.334314 25186 ts_tablet_manager.cc:1905] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:28.334326 25187 ts_tablet_manager.cc:1905] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:06:28.335920 25186 ts_tablet_manager.cc:1918] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:28.335920 25187 ts_tablet_manager.cc:1918] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:06:28.335994 25186 log.cc:1196] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/17d31016fbaa4e9e9460838e5ed90894
I20260925 05:06:28.336084 25187 log.cc:1196] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/17d31016fbaa4e9e9460838e5ed90894
I20260925 05:06:28.336277 25186 ts_tablet_manager.cc:1939] T 17d31016fbaa4e9e9460838e5ed90894 P 0862fe55f98545a68ae6013ff7c912b8: Deleting consensus metadata
I20260925 05:06:28.336447 25187 ts_tablet_manager.cc:1939] T 17d31016fbaa4e9e9460838e5ed90894 P bf9d005af9b148b6a40c35f821ae61cd: Deleting consensus metadata
I20260925 05:06:28.336655 24676 catalog_manager.cc:4841] TS 0862fe55f98545a68ae6013ff7c912b8 (127.14.33.3:33875): tablet 17d31016fbaa4e9e9460838e5ed90894 (table test-table [id=a8349a289c5049fd9c539661b935dd92]) successfully deleted
I20260925 05:06:28.336876 24674 catalog_manager.cc:4841] TS bf9d005af9b148b6a40c35f821ae61cd (127.14.33.2:33489): tablet 17d31016fbaa4e9e9460838e5ed90894 (table test-table [id=a8349a289c5049fd9c539661b935dd92]) successfully deleted
W20260925 05:06:28.338521 24676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:41361 (error 108)
I20260925 05:06:28.338706 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24887
I20260925 05:06:28.338954 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:06:28.338858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:41361 (local address 127.0.0.1:41722)
0925 05:06:28.338897 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 05:06:28.338924 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 05:06:28.339043 24676 catalog_manager.cc:4584] TS d10e49eea7494a4f97506d4e4bef0103 (127.14.33.1:41361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17d31016fbaa4e9e9460838e5ed90894: Network error: Client connection negotiation failed: client connection to 127.14.33.1:41361: connect: Connection refused (error 111)
I20260925 05:06:28.343381 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 25021
I20260925 05:06:28.348097 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 24658
2026-09-25T05:06:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1775 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[25192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[25192](info): set up 2 sockets
Sep 25 05:06:28 dist-test-slave-4dmq krb5kdc[25192](info): commencing operation
krb5kdc: starting...
W20260925 05:06:29.318558 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:06:29 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:06:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/rangeradmin_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/rangeradmin_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/rangerlookup_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/rangerlookup_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
I20260925 05:06:29.461376 14468 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task85DeA_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/postgres -l logfile start
2026-09-25 05:06:31.632 UTC [25222] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:06:31.632 UTC [25222] LOG: listening on IPv4 address "127.14.33.20", port 32999
2026-09-25 05:06:31.635 UTC [25222] LOG: listening on Unix socket "/tmp/.s.PGSQL.32999"
2026-09-25 05:06:31.670 UTC [25225] LOG: database system was shut down at 2026-09-25 05:06:30 UTC
2026-09-25 05:06:31.674 UTC [25222] LOG: database system is ready to accept connections
I20260925 05:06:32.555028 14468 mini_postgres.cc:96] Postgres bound to 32999
2026-09-25 05:06:32.561 UTC [25233] FATAL: database "slave" does not exist
127.14.33.20:32999 - accepting connections
I20260925 05:06:32.561555 14468 mini_ranger.cc:161] Starting Ranger...
I20260925 05:06:32.575047 14468 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:06:32.972918 14468 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:06:32.973001 14468 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-admin
2026-09-25 05:06:33,239 [I] DB FLAVOR :POSTGRES
2026-09-25 05:06:33,239 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:06:33,239 [I] Checking connection..
2026-09-25 05:06:33,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:06:33,451 [I] Checking connection passed.
2026-09-25 05:06:33,451 [I] --------- Verifying version history table ---------
2026-09-25 05:06:33,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:06:33,677 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:06:33,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:06:33,896 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:06:33,896 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:06:33,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:06:34.112 UTC [25297] WARNING: there is no transaction in progress
2026-09-25 05:06:34,117 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:06:34,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:06:34,346 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:06:34,347 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:06:34,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:06:34,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:06:34,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4dmq.c.gcp-upstream.internal','N') ;"
2026-09-25 05:06:35,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:06:35,304 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:06:35,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:06:35,517 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:06:35,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:06:35,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:06:35,973 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:06:35,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:06:36.855 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.869 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.885 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.902 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.918 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.938 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.990 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:36.996 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.010 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.019 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.028 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.042 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.051 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.057 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.063 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37.069 UTC [25405] WARNING: there is no transaction in progress
2026-09-25 05:06:37,680 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:06:37,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:06:37,907 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:06:37,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:06:38,125 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:06:38,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:06:38,367 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:06:38,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:06:38,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:06:38,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4dmq.c.gcp-upstream.internal';"
2026-09-25 05:06:39,055 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:06:39,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:06:39,278 [I] DB_PATCHES have already been applied
I20260925 05:06:39.283953 14468 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-admin:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task85DeA_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:06:39.284035 14468 mini_ranger.cc:193] Using host: 127.14.33.20
I20260925 05:06:39.286855 14468 mini_ranger.cc:227] Ranger admin URL: http://127.14.33.20:48835
Sep 25, 2026 5:06:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:06:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:06:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:06:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:06:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:06:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:06:40.322113 14468 mini_ranger.cc:160] Time spent starting Ranger: real 7.761s user 0.006s sys 0.001s
Sep 25, 2026 5:06:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:06:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 5:06:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:06:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:06:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:06:59.506063 14468 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/rangerkms_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/rangerkms_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.33.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.33.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:06:59 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312819, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:06:59.580394 14468 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:06:59.589052 14468 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:07:00.354477 14468 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:07:00.354581 14468 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms
I20260925 05:07:00.354605 14468 mini_ranger_kms.cc:227] Using postgres at 127.14.33.20:32999
2026-09-25 05:07:00,511 [I] DB FLAVOR :POSTGRES
2026-09-25 05:07:00,511 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:07:00,511 [I] Checking connection
2026-09-25 05:07:00,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:07:00,731 [I] connection success
2026-09-25 05:07:00,731 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:07:00,732 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:07:00,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:07:00,940 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:07:00,940 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:07:00,940 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:07:00,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:32999/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:07:01,189 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:07:01.639528 14468 mini_ranger_kms.cc:310] Created kms service
I20260925 05:07:01.836398 14468 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:07:01.878674 14468 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:07:01 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312819, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:02.243566 14468 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:07:02.243698 14468 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms:/tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:07:02.243732 14468 mini_ranger_kms.cc:240] Using host: 127.14.33.20
I20260925 05:07:02.247546 14468 mini_ranger_kms.cc:276] Ranger KMS PID: 25585
I20260925 05:07:02.247664 14468 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.33.20:54367
Sep 25, 2026 5:07:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:07:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:07:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:07:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:07:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:07:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 05:07:04 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312824, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.33.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:04.423259 14468 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.843s user 0.000s sys 0.141s
I20260925 05:07:04.423359 14468 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:07:04.423452 14468 mini_ranger_kms.cc:401] 127.14.33.20:54367/kms/v1/keys
Sep 25 05:07:04 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312824, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.33.20@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
Sep 25 05:07:05 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312824, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.33.20@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
Sep 25 05:07:07 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:07:07.251434 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41233
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41233
--ranger_config_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:54367/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:07.331851 25636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:07.332032 25636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:07.332053 25636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:07.333247 25636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:07.333287 25636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:07.333302 25636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:07.333314 25636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:07.333326 25636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:07.334867 25636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.33.20:54367/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41233
--ranger_config_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:07.335090 25636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:07.335335 25636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:07.340471 25644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:07.340471 25641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:07.340482 25642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:07.340727 25636 server_base.cc:1034] running on GCE node
I20260925 05:07:07.340969 25636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:07.341285 25636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:07.342419 25636 hybrid_clock.cc:648] HybridClock initialized: now 1790312827342409 us; error 25 us; skew 500 ppm
Sep 25 05:07:07 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:07.344880 25636 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:07:07.345345 25636 webserver.cc:460] Webserver started at http://127.14.33.62:45749/ using document root <none> and password file <none>
I20260925 05:07:07.345511 25636 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:07.345546 25636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:07.345624 25636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:07 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:07.402812 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "1f800c9bd949433cbb454a58574685b2"
format_stamp: "Formatted at 2026-09-25 05:07:07 on dist-test-slave-4dmq"
server_key: "b97822811486b91055a85515f6572806"
server_key_iv: "417ce9d16211de7aec0a0c2dd834e845"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:07.403270 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "1f800c9bd949433cbb454a58574685b2"
format_stamp: "Formatted at 2026-09-25 05:07:07 on dist-test-slave-4dmq"
server_key: "b97822811486b91055a85515f6572806"
server_key_iv: "417ce9d16211de7aec0a0c2dd834e845"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:07.405109 25636 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 05:07:07.419700 25651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:07.419901 25636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:07:07.419996 25636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "1f800c9bd949433cbb454a58574685b2"
format_stamp: "Formatted at 2026-09-25 05:07:07 on dist-test-slave-4dmq"
server_key: "b97822811486b91055a85515f6572806"
server_key_iv: "417ce9d16211de7aec0a0c2dd834e845"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:07.420092 25636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:07.455292 25636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:07.456982 25636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:07.457088 25636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:07.461282 25636 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:41233
I20260925 05:07:07.461345 25703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:41233 every 8 connection(s)
I20260925 05:07:07.461669 25636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:07.462302 25704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:07.464558 25704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2: Bootstrap starting.
I20260925 05:07:07.465229 25704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:07.465567 25704 log.cc:824] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:07.466216 25704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2: No bootstrap required, opened a new log
I20260925 05:07:07.466867 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 25636
I20260925 05:07:07.466986 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:07:07.467154 14468 external_mini_cluster.cc:1428] Setting key 935208ab3eac933a7f827f3fdc7d022c
I20260925 05:07:07.468565 25704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f800c9bd949433cbb454a58574685b2" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41233 } }
I20260925 05:07:07.468730 25704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:07.468793 25704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f800c9bd949433cbb454a58574685b2, State: Initialized, Role: FOLLOWER
I20260925 05:07:07.468941 25704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f800c9bd949433cbb454a58574685b2" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41233 } }
I20260925 05:07:07.469005 25704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:07.469027 25704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:07.469079 25704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:07.469615 25704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f800c9bd949433cbb454a58574685b2" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41233 } }
I20260925 05:07:07.469732 25704 leader_election.cc:302] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f800c9bd949433cbb454a58574685b2; no voters:
I20260925 05:07:07.469907 25704 leader_election.cc:288] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:07.469938 25709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:07.470037 25709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [term 1 LEADER]: Becoming Leader. State: Replica: 1f800c9bd949433cbb454a58574685b2, State: Running, Role: LEADER
I20260925 05:07:07.470152 25704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:07.470191 25709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f800c9bd949433cbb454a58574685b2" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41233 } }
I20260925 05:07:07.470589 25711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f800c9bd949433cbb454a58574685b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f800c9bd949433cbb454a58574685b2" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41233 } } }
I20260925 05:07:07.470676 25711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:07.470912 25710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f800c9bd949433cbb454a58574685b2. Latest consensus state: current_term: 1 leader_uuid: "1f800c9bd949433cbb454a58574685b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f800c9bd949433cbb454a58574685b2" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41233 } } }
I20260925 05:07:07.470990 25710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:07.471652 25704 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:07:07.473748 25716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:07.474882 25716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:07:07 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:07.477651 25716 catalog_manager.cc:1347] Generated new cluster ID: 314fa4c533b048bc9a57fad817fc35ce
I20260925 05:07:07.477712 25716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:07.480962 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:07.467684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.14.33.62:41233)
0925 05:07:07.467852 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:07.467862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:07.467910 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:07:07.468558 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:07.468565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:07.468610 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:07.468806 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:07.469489 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:07.470716 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:07.471604 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:07.474225 (+ 2621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:07.474422 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:07.477383 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:07.477387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:07.477387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:07.477400 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:07.479266 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:07.479618 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:07.479621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:07.479621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:07.479766 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:07.479966 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:07.479967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:07.479967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:07.480313 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:07.480475 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:07.480565 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 05:07:07.480715 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260925 05:07:07.504443 25716 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:07.505004 25716 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:07.509774 25716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2: Generated new TSK 0
I20260925 05:07:07.509949 25716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:07:08.165117 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:54367/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41233
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:08.285322 25749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:08.285616 25749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:08.285665 25749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:08.287335 25749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:08.287395 25749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:08.287432 25749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:07:08.289539 25749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.33.20:54367/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:08.289894 25749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:08.290190 25749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:08.295876 25755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.295876 25754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.296068 25757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:08.295975 25749 server_base.cc:1034] running on GCE node
I20260925 05:07:08.296335 25749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:08.296617 25749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:08.297803 25749 hybrid_clock.cc:648] HybridClock initialized: now 1790312828297783 us; error 34 us; skew 500 ppm
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:08.300765 25749 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:07:08.301296 25749 webserver.cc:460] Webserver started at http://127.14.33.1:37047/ using document root <none> and password file <none>
I20260925 05:07:08.301539 25749 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:08.301618 25749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:08.301802 25749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:08.316110 25749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "1b177bc6314d4e82a652041d05ef6be5"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "8e08474defd0e347ae10bc0c9d7732b5"
server_key_iv: "f875f1f00ef0b2d5a8dfa4316ddcb8e7"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.316860 25749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "1b177bc6314d4e82a652041d05ef6be5"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "8e08474defd0e347ae10bc0c9d7732b5"
server_key_iv: "f875f1f00ef0b2d5a8dfa4316ddcb8e7"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.321414 25749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260925 05:07:08.334295 25764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.334909 25749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:08.335027 25749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "1b177bc6314d4e82a652041d05ef6be5"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "8e08474defd0e347ae10bc0c9d7732b5"
server_key_iv: "f875f1f00ef0b2d5a8dfa4316ddcb8e7"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.335117 25749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:08.361429 25749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:08.363931 25749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:08.364104 25749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:08.364369 25749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:08.364711 25749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:08.364773 25749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.364807 25749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:08.364833 25749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.372617 25749 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39775
I20260925 05:07:08.372975 25749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:08.373566 25879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39775 every 8 connection(s)
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:08.379940 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 25749
I20260925 05:07:08.380053 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:08.380241 14468 external_mini_cluster.cc:1428] Setting key a4226d67c5fac96d843a9626b75d189f
I20260925 05:07:08.382809 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.373770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41233 (local address 127.14.33.1:47647)
0925 05:07:08.374065 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.374097 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.374603 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.374930 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.374933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.375112 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:08.375303 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.375307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.376005 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.376006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.376818 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.376820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.377270 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.377271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.377389 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.377897 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:08.377905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:08.379132 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:08.381247 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.381253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.381256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.381563 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.381781 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.381782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.381782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.381915 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.382198 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:08.382204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:08.382394 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.382509 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.382651 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":74,"threads_started":1}
I20260925 05:07:08.383244 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.373780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:47647 (local address 127.14.33.62:41233)
0925 05:07:08.374006 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.374009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.374188 (+ 179us) server_negotiation.cc:407] Connection header received
0925 05:07:08.374710 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.374712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.374726 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.374836 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:08.375398 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.375911 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.376925 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.377184 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.377254 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.379254 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:08.379257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:08.379257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:08.379264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:08.381149 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.381648 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.381649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.381650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.381715 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.381979 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.381980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.381980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.382119 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:08.382174 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.383062 (+ 888us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.383143 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260925 05:07:08.383697 25880 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41233
I20260925 05:07:08.383798 25880 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:08.384013 25880 heartbeater.cc:511] Master 127.14.33.62:41233 requested a full tablet report, sending...
I20260925 05:07:08.384619 25668 ts_manager.cc:194] Registered new tserver with Master: 1b177bc6314d4e82a652041d05ef6be5 (127.14.33.1:39775)
I20260925 05:07:08.385231 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:47647
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:07:08.413790 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:54367/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41233
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:08.513826 25886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:08.514093 25886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:08.514142 25886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:08.515542 25886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:08.515615 25886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:08.515659 25886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:07:08.517308 25886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.33.20:54367/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:08.517666 25886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:08.517931 25886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:08.523447 25893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.523604 25896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.523864 25894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:08.524227 25886 server_base.cc:1034] running on GCE node
I20260925 05:07:08.524458 25886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:08.524744 25886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:08.525890 25886 hybrid_clock.cc:648] HybridClock initialized: now 1790312828525872 us; error 33 us; skew 500 ppm
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:08.528685 25886 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:07:08.529188 25886 webserver.cc:460] Webserver started at http://127.14.33.2:33201/ using document root <none> and password file <none>
I20260925 05:07:08.529402 25886 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:08.529469 25886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:08.529598 25886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:08.548859 25886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "1cbc8e11da6f4664956f0c5be5fd09fa"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "abaccb8a94cc52f27eb0edc26fede560"
server_key_iv: "c67bfe3288d70072897ee4f46090a366"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.549366 25886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "1cbc8e11da6f4664956f0c5be5fd09fa"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "abaccb8a94cc52f27eb0edc26fede560"
server_key_iv: "c67bfe3288d70072897ee4f46090a366"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.551120 25886 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 05:07:08.573227 25903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.573467 25886 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260925 05:07:08.573553 25886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "1cbc8e11da6f4664956f0c5be5fd09fa"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "abaccb8a94cc52f27eb0edc26fede560"
server_key_iv: "c67bfe3288d70072897ee4f46090a366"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.573637 25886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:08.614563 25886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:08.616957 25886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:08.617098 25886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:08.617411 25886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:08.617789 25886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:08.617861 25886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.617918 25886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:08.617959 25886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.624619 25886 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:41641
I20260925 05:07:08.625080 25886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:08.626497 26017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:41641 every 8 connection(s)
I20260925 05:07:08.629793 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 25886
I20260925 05:07:08.629899 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:08.630062 14468 external_mini_cluster.cc:1428] Setting key 8186e1a0bee678d8549ac7e845c7cf4a
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:08.638447 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.627504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:44241 (local address 127.14.33.62:41233)
0925 05:07:08.627583 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.627585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.627749 (+ 164us) server_negotiation.cc:407] Connection header received
0925 05:07:08.628376 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.628377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.628389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.628462 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:08.629178 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.629945 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.631309 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.631534 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.631630 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.634944 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:08.634946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:08.634946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:08.634955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:08.636684 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.637267 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.637269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.637269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.637334 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.637662 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.637663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.637663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.637800 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:08.637866 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.638227 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.638333 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:07:08.639266 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.627432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41233 (local address 127.14.33.2:44241)
0925 05:07:08.627607 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.627649 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.628252 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.628534 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.628539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.628760 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:08.629026 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.629032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.630070 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.630071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.631201 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.631204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.631630 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.631631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.631754 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.632248 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:08.632262 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:08.634841 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:08.636774 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.636781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.636783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.637136 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.637417 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.637418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.637419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.637572 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.637874 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:08.637879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:08.638053 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.638803 (+ 750us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.638964 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 05:07:08.639916 26018 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41233
I20260925 05:07:08.640008 26018 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:08.640185 26018 heartbeater.cc:511] Master 127.14.33.62:41233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
I20260925 05:07:08.640618 25668 ts_manager.cc:194] Registered new tserver with Master: 1cbc8e11da6f4664956f0c5be5fd09fa (127.14.33.2:41641)
I20260925 05:07:08.641052 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:44241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:07:08.656301 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:54367/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41233
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:08.766858 26023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:08.767040 26023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:08.767104 26023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:08.768214 26023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:08.768280 26023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:08.768308 26023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:07:08.769785 26023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.33.20:54367/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:08.769987 26023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:08.770267 26023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:08.775220 26031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.775486 26029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:08.775609 26028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:08.775728 26023 server_base.cc:1034] running on GCE node
I20260925 05:07:08.775944 26023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:08.776180 26023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:08.777333 26023 hybrid_clock.cc:648] HybridClock initialized: now 1790312828777324 us; error 33 us; skew 500 ppm
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:08.779794 26023 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:07:08.780185 26023 webserver.cc:460] Webserver started at http://127.14.33.3:44929/ using document root <none> and password file <none>
I20260925 05:07:08.780395 26023 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:08.780460 26023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:08.780647 26023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:08.795889 26023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "1f902d4dddcd4c40b18586265084afb0"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "5282c6fb402e70333616e35f470ce262"
server_key_iv: "8c55c4573ebb5a4d9e179b98e2c97fad"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.796303 26023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "1f902d4dddcd4c40b18586265084afb0"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "5282c6fb402e70333616e35f470ce262"
server_key_iv: "8c55c4573ebb5a4d9e179b98e2c97fad"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.798116 26023 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260925 05:07:08.821305 26038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.821527 26023 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 05:07:08.821661 26023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "1f902d4dddcd4c40b18586265084afb0"
format_stamp: "Formatted at 2026-09-25 05:07:08 on dist-test-slave-4dmq"
server_key: "5282c6fb402e70333616e35f470ce262"
server_key_iv: "8c55c4573ebb5a4d9e179b98e2c97fad"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:08.821837 26023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:08.842814 26023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:08.844841 26023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:08.845000 26023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:08.845278 26023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:08.845631 26023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:08.845681 26023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.845759 26023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:08.845788 26023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:08.852499 26023 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:33653
I20260925 05:07:08.852741 26151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:33653 every 8 connection(s)
I20260925 05:07:08.852969 26023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:08.862089 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 26023
I20260925 05:07:08.862186 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:08.862365 14468 external_mini_cluster.cc:1428] Setting key 78a8ecd16a045a191c3cc9756d26c848
I20260925 05:07:08.864820 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.855121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:42131 (local address 127.14.33.62:41233)
0925 05:07:08.855157 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.855159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.855168 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:07:08.855718 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.855719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.855732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.855791 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:08.856380 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.856883 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.857954 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.858309 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.858392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.860966 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:08.860967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:08.860968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:08.860975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:08.863354 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.863827 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.863828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.863828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.863901 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.864191 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.864192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.864192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.864314 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:08.864389 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.864656 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.864731 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:07:08.865018 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.854844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41233 (local address 127.14.33.3:42131)
0925 05:07:08.854981 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.855031 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.855570 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.855917 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.855921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.856107 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:08.856287 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.856290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.856997 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.856998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.857864 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.857867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.858400 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.858402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.858529 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.859002 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:08.859010 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:08.860176 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:08.863436 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.863442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.863444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.863746 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.863975 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:08.863976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:08.863977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:08.864113 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:08.864379 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:08.864385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:08.864528 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.864637 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.864785 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":78,"threads_started":1}
I20260925 05:07:08.865613 26152 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41233
I20260925 05:07:08.865701 26152 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:08.865880 26152 heartbeater.cc:511] Master 127.14.33.62:41233 requested a full tablet report, sending...
I20260925 05:07:08.866253 25668 ts_manager.cc:194] Registered new tserver with Master: 1f902d4dddcd4c40b18586265084afb0 (127.14.33.3:33653)
I20260925 05:07:08.866673 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:42131
I20260925 05:07:08.874635 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:08.883132 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.876085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.14.33.62:41233)
0925 05:07:08.876122 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.876124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.876287 (+ 163us) server_negotiation.cc:407] Connection header received
0925 05:07:08.876361 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.876362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.876372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.876438 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:08.877062 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.877601 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.878626 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.878896 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.878936 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.880330 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:08.880331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:08.880331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:08.880338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:08.881999 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.882383 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.882384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.882385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.882437 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:08.882640 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:08.882641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:08.882641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:08.882752 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:08.882853 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.882964 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.883031 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:07:08.884827 25668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:08.886479 25668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:08.893992 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.888710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:41641 (local address 127.0.0.1:52968)
0925 05:07:08.889148 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.889190 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.889419 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.890239 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.890240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.890242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:08.890275 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.890276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.891394 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.891395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.892612 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.892615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.893640 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.893640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.893720 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.893747 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.893811 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.893914 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260925 05:07:08.894125 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.888712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39775 (local address 127.0.0.1:42980)
0925 05:07:08.889656 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.889665 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.889709 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.889949 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.889951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.889953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:08.890013 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.890016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.891264 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.891265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.892612 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.892615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.893635 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.893636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.893958 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.893962 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.894009 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.894045 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260925 05:07:08.894317 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.888964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.14.33.1:39775)
0925 05:07:08.889374 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.889379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.889782 (+ 403us) server_negotiation.cc:407] Connection header received
0925 05:07:08.889817 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.889819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.889834 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.889925 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:08.890092 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.891168 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.892710 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.893459 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.893525 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.893596 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.894181 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.894225 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":47,"threads_started":1}
I20260925 05:07:08.895076 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.889225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.14.33.2:41641)
0925 05:07:08.890039 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.890043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.890053 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:07:08.890090 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.890092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.890106 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.890175 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:08.890347 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.891317 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.892729 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.893457 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.893526 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.893612 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.894961 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.895003 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":81,"threads_started":1}
I20260925 05:07:08.895251 25814 tablet_service.cc:1467] Processing CreateTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae (DEFAULT_TABLE table=test-table [id=176bb4df2c6347fa82e9103a2639fea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:08.895635 25814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f7ec1f2f9c4197b34aa9601363dcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:08.896741 25952 tablet_service.cc:1467] Processing CreateTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae (DEFAULT_TABLE table=test-table [id=176bb4df2c6347fa82e9103a2639fea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:08.897068 25952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f7ec1f2f9c4197b34aa9601363dcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:08.897773 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.888929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:33653 (local address 127.0.0.1:37906)
0925 05:07:08.894618 (+ 5689us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:08.894629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:08.894670 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:08.894870 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:08.894871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:08.894873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:08.894898 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.894900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.895898 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.895899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.896670 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:08.896672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.897560 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.897561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:08.897620 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.897623 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:07:08.897677 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:07:08.897711 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5677,"thread_start_us":58,"threads_started":1}
I20260925 05:07:08.898036 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:08.889406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.14.33.3:33653)
0925 05:07:08.889607 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:08.889611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:08.894726 (+ 5115us) server_negotiation.cc:407] Connection header received
0925 05:07:08.894776 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:08.894779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:08.894793 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:08.894859 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:08.894968 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.895815 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.896752 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:08.897384 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:08.897454 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:08.897526 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:08.897916 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 05:07:08.897957 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260925 05:07:08.898641 26067 tablet_service.cc:1467] Processing CreateTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae (DEFAULT_TABLE table=test-table [id=176bb4df2c6347fa82e9103a2639fea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:08.898945 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f7ec1f2f9c4197b34aa9601363dcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:08.901297 26170 tablet_bootstrap.cc:492] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: Bootstrap starting.
I20260925 05:07:08.902251 26172 tablet_bootstrap.cc:492] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Bootstrap starting.
I20260925 05:07:08.902566 26170 tablet_bootstrap.cc:654] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:08.902863 26170 log.cc:824] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:08.903187 26172 tablet_bootstrap.cc:654] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:08.903623 26172 log.cc:824] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:08.904399 26172 tablet_bootstrap.cc:492] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: No bootstrap required, opened a new log
I20260925 05:07:08.904464 26172 ts_tablet_manager.cc:1397] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:07:08.906345 26172 raft_consensus.cc:348] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:08.906497 26172 raft_consensus.cc:374] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:08.906527 26172 raft_consensus.cc:729] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f902d4dddcd4c40b18586265084afb0, State: Initialized, Role: FOLLOWER
I20260925 05:07:08.906630 26172 consensus_queue.cc:260] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:08.906848 26172 ts_tablet_manager.cc:1428] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:07:08.907032 26152 heartbeater.cc:503] Master 127.14.33.62:41233 was elected leader, sending a full tablet report...
I20260925 05:07:08.909302 26171 tablet_bootstrap.cc:492] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: Bootstrap starting.
I20260925 05:07:08.909747 26170 tablet_bootstrap.cc:492] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: No bootstrap required, opened a new log
I20260925 05:07:08.909871 26170 ts_tablet_manager.cc:1397] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 05:07:08.910274 26171 tablet_bootstrap.cc:654] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:08.910570 26171 log.cc:824] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:08.912370 26170 raft_consensus.cc:348] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:08.912554 26170 raft_consensus.cc:374] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:08.912618 26170 raft_consensus.cc:729] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b177bc6314d4e82a652041d05ef6be5, State: Initialized, Role: FOLLOWER
I20260925 05:07:08.912751 26170 consensus_queue.cc:260] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:08.913333 26170 ts_tablet_manager.cc:1428] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:08.913484 25880 heartbeater.cc:503] Master 127.14.33.62:41233 was elected leader, sending a full tablet report...
I20260925 05:07:08.917721 26171 tablet_bootstrap.cc:492] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: No bootstrap required, opened a new log
I20260925 05:07:08.917790 26171 ts_tablet_manager.cc:1397] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260925 05:07:08.919586 26171 raft_consensus.cc:348] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:08.919790 26171 raft_consensus.cc:374] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:08.919839 26171 raft_consensus.cc:729] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbc8e11da6f4664956f0c5be5fd09fa, State: Initialized, Role: FOLLOWER
I20260925 05:07:08.919942 26171 consensus_queue.cc:260] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:08.920188 26171 ts_tablet_manager.cc:1428] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:07:08.920224 26018 heartbeater.cc:503] Master 127.14.33.62:41233 was elected leader, sending a full tablet report...
Sep 25 05:07:08 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
W20260925 05:07:09.104602 26153 tablet.cc:2369] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:07:09.124027 25881 tablet.cc:2369] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:07:09.127636 26019 tablet.cc:2369] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:09.140991 26178 raft_consensus.cc:482] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:09.141150 26178 raft_consensus.cc:504] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:09.141577 26178 leader_election.cc:288] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b177bc6314d4e82a652041d05ef6be5 (127.14.33.1:39775), 1f902d4dddcd4c40b18586265084afb0 (127.14.33.3:33653)
I20260925 05:07:09.145401 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.141972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:43897 (local address 127.14.33.3:33653)
0925 05:07:09.142011 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:09.142013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:09.142155 (+ 142us) server_negotiation.cc:407] Connection header received
0925 05:07:09.142257 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:09.142258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:09.142268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:09.142295 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:09.142487 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.143174 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.144534 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.145027 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.145073 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.145084 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:09.145300 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 05:07:09.145338 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:07:09.145505 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.141923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:59521 (local address 127.14.33.1:39775)
0925 05:07:09.141965 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:09.141967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:09.142132 (+ 165us) server_negotiation.cc:407] Connection header received
0925 05:07:09.142240 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:09.142242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:09.142254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:09.142284 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:09.142516 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.143231 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.144547 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.145145 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.145190 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.145199 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:09.145403 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 05:07:09.145442 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:07:09.145694 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.141759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39775 (local address 127.14.33.2:59521)
0925 05:07:09.142051 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:09.142061 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:09.142163 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:09.142356 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:09.142357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:09.142358 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:09.142432 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.142435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.143325 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.143326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.144447 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.144450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.145256 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.145257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.145327 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.145330 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:07:09.145587 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 05:07:09.145627 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":15,"thread_start_us":65,"threads_started":1}
I20260925 05:07:09.146015 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.141801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:33653 (local address 127.14.33.2:43897)
0925 05:07:09.142063 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:09.142072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:09.142184 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:09.142353 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:09.142354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:09.142356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:09.142426 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.142428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.143281 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.143282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.144419 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:09.144422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.145165 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.145166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:09.145226 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.145230 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:07:09.145897 (+ 667us) client_negotiation.cc:240] Negotiation successful
0925 05:07:09.145938 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260925 05:07:09.146036 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f7ec1f2f9c4197b34aa9601363dcae" candidate_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b177bc6314d4e82a652041d05ef6be5" is_pre_election: true
I20260925 05:07:09.146186 25834 raft_consensus.cc:2454] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbc8e11da6f4664956f0c5be5fd09fa in term 0.
I20260925 05:07:09.146334 26106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f7ec1f2f9c4197b34aa9601363dcae" candidate_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f902d4dddcd4c40b18586265084afb0" is_pre_election: true
I20260925 05:07:09.146414 25905 leader_election.cc:302] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b177bc6314d4e82a652041d05ef6be5, 1cbc8e11da6f4664956f0c5be5fd09fa; no voters:
I20260925 05:07:09.146462 26106 raft_consensus.cc:2454] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbc8e11da6f4664956f0c5be5fd09fa in term 0.
I20260925 05:07:09.146538 26178 raft_consensus.cc:2781] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:09.146579 26178 raft_consensus.cc:482] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:09.146600 26178 raft_consensus.cc:3037] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:09.147485 26178 raft_consensus.cc:504] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:09.147600 26178 leader_election.cc:288] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [CANDIDATE]: Term 1 election: Requested vote from peers 1b177bc6314d4e82a652041d05ef6be5 (127.14.33.1:39775), 1f902d4dddcd4c40b18586265084afb0 (127.14.33.3:33653)
I20260925 05:07:09.148057 26106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f7ec1f2f9c4197b34aa9601363dcae" candidate_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f902d4dddcd4c40b18586265084afb0"
I20260925 05:07:09.148130 26106 raft_consensus.cc:3037] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:09.149010 26106 raft_consensus.cc:2454] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbc8e11da6f4664956f0c5be5fd09fa in term 1.
I20260925 05:07:09.149199 25907 leader_election.cc:302] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbc8e11da6f4664956f0c5be5fd09fa, 1f902d4dddcd4c40b18586265084afb0; no voters:
I20260925 05:07:09.149369 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f7ec1f2f9c4197b34aa9601363dcae" candidate_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b177bc6314d4e82a652041d05ef6be5"
I20260925 05:07:09.149441 25834 raft_consensus.cc:3037] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:09.150204 26178 raft_consensus.cc:2781] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:09.150313 25834 raft_consensus.cc:2454] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbc8e11da6f4664956f0c5be5fd09fa in term 1.
I20260925 05:07:09.150354 26178 raft_consensus.cc:686] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 1 LEADER]: Becoming Leader. State: Replica: 1cbc8e11da6f4664956f0c5be5fd09fa, State: Running, Role: LEADER
I20260925 05:07:09.150445 26178 consensus_queue.cc:237] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } }
I20260925 05:07:09.151228 25668 catalog_manager.cc:5495] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbc8e11da6f4664956f0c5be5fd09fa (127.14.33.2). New cstate: current_term: 1 leader_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbc8e11da6f4664956f0c5be5fd09fa" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 41641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260925 05:07:09.177587 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.174439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.14.33.2:41641)
0925 05:07:09.174479 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:09.174481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:09.174488 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:07:09.174527 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:09.174528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:09.174549 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:09.174628 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:09.174780 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.175481 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.176537 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:09.176800 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:09.176854 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:09.177033 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:09.177186 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:09.177257 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:09.177477 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 05:07:09.177497 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:07:09.183285 26106 raft_consensus.cc:1261] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 1 FOLLOWER]: Refusing update from remote peer 1cbc8e11da6f4664956f0c5be5fd09fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:09.183331 25834 raft_consensus.cc:1261] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 1 FOLLOWER]: Refusing update from remote peer 1cbc8e11da6f4664956f0c5be5fd09fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:09.183748 26181 consensus_queue.cc:1035] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f902d4dddcd4c40b18586265084afb0" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:09.183874 26178 consensus_queue.cc:1035] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b177bc6314d4e82a652041d05ef6be5" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 05:07:09 dist-test-slave-4dmq krb5kdc[25192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312828, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:09.189265 26187 mvcc.cc:205] Tried to move back new op lower bound from 7333121348331274240 to 7333121348200796160. Current Snapshot: MvccSnapshot[applied={T|T < 7333121348331274240}]
I20260925 05:07:09.190075 25668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:09.190172 25668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:09.195269 26186 mvcc.cc:205] Tried to move back new op lower bound from 7333121348331274240 to 7333121348200796160. Current Snapshot: MvccSnapshot[applied={T|T < 7333121348331274240}]
I20260925 05:07:09.196642 25668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f800c9bd949433cbb454a58574685b2: Sending DeleteTablet for 3 replicas of tablet b7f7ec1f2f9c4197b34aa9601363dcae
I20260925 05:07:09.196947 26067 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:09 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260925 05:07:09.197027 25952 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:09 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:52968
I20260925 05:07:09.197140 26068 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:09 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:37906
I20260925 05:07:09.197258 25814 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f7ec1f2f9c4197b34aa9601363dcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:09 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42980
I20260925 05:07:09.197490 26196 tablet_replica.cc:330] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5: stopping tablet replica
I20260925 05:07:09.197598 26196 raft_consensus.cc:2229] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:09.197671 26196 raft_consensus.cc:2258] T b7f7ec1f2f9c4197b34aa9601363dcae P 1b177bc6314d4e82a652041d05ef6be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:09.197829 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 25749
I20260925 05:07:09.205032 26195 tablet_replica.cc:330] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa: stopping tablet replica
I20260925 05:07:09.205170 26195 raft_consensus.cc:2229] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:09.205277 26195 raft_consensus.cc:2258] T b7f7ec1f2f9c4197b34aa9601363dcae P 1cbc8e11da6f4664956f0c5be5fd09fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 05:07:09.205339 25653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:39775 (error 108)
I20260925 05:07:09.205461 26194 tablet_replica.cc:330] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: stopping tablet replica
I20260925 05:07:09.205456 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 25886
I20260925 05:07:09.206872 26194 raft_consensus.cc:2229] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:09.206972 26194 raft_consensus.cc:2258] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:09.208941 26194 ts_tablet_manager.cc:1905] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:09.210743 26194 ts_tablet_manager.cc:1918] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:09.210827 26194 log.cc:1196] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/b7f7ec1f2f9c4197b34aa9601363dcae
I20260925 05:07:09.211090 26194 ts_tablet_manager.cc:1939] T b7f7ec1f2f9c4197b34aa9601363dcae P 1f902d4dddcd4c40b18586265084afb0: Deleting consensus metadata
I20260925 05:07:09.211524 25655 catalog_manager.cc:4841] TS 1f902d4dddcd4c40b18586265084afb0 (127.14.33.3:33653): tablet b7f7ec1f2f9c4197b34aa9601363dcae (table test-table [id=176bb4df2c6347fa82e9103a2639fea2]) successfully deleted
W20260925 05:07:09.211766 25655 catalog_manager.cc:4808] TS 1f902d4dddcd4c40b18586265084afb0 (127.14.33.3:33653): delete failed for tablet b7f7ec1f2f9c4197b34aa9601363dcae because the tablet was not found. No further retry: Not found: Tablet not found: b7f7ec1f2f9c4197b34aa9601363dcae
I20260925 05:07:09.213531 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.213415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39775 (local address 127.0.0.1:42994)
0925 05:07:09.213462 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:09.213497 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:07:09.213634 25653 catalog_manager.cc:4584] TS 1b177bc6314d4e82a652041d05ef6be5 (127.14.33.1:39775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7f7ec1f2f9c4197b34aa9601363dcae: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39775: connect: Connection refused (error 111)
I20260925 05:07:09.215585 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 26023
I20260925 05:07:09.215605 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:09.215475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:41641 (local address 127.0.0.1:52986)
0925 05:07:09.215540 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:09.215574 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.2:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 05:07:09.222172 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 25636
I20260925 05:07:09.229722 14468 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:07:09.574524 14468 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:07:09.574677 14468 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:07:09.918671 14468 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:07:09.918 UTC [25222] LOG: received smart shutdown request
2026-09-25 05:07:09.921 UTC [25222] LOG: background worker "logical replication launcher" (PID 25231) exited with exit code 1
2026-09-25 05:07:09.922 UTC [25226] LOG: shutting down
2026-09-25 05:07:10.065 UTC [25222] LOG: database system is shut down
2026-09-25T05:07:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41817 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:10 dist-test-slave-4dmq krb5kdc[26207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:10 dist-test-slave-4dmq krb5kdc[26207](info): set up 2 sockets
Sep 25 05:07:10 dist-test-slave-4dmq krb5kdc[26207](info): commencing operation
krb5kdc: starting...
W20260925 05:07:11.161850 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:07:11 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/rangeradmin_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/rangeradmin_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/rangerlookup_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/rangerlookup_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
I20260925 05:07:11.309918 14468 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task85DeA_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/postgres -l logfile start
2026-09-25 05:07:13.616 UTC [26237] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 05:07:13.616 UTC [26237] LOG: listening on IPv4 address "127.14.33.20", port 42543
2026-09-25 05:07:13.620 UTC [26237] LOG: listening on Unix socket "/tmp/.s.PGSQL.42543"
2026-09-25 05:07:13.640 UTC [26240] LOG: database system was shut down at 2026-09-25 05:07:12 UTC
2026-09-25 05:07:13.644 UTC [26237] LOG: database system is ready to accept connections
I20260925 05:07:14.598069 14468 mini_postgres.cc:96] Postgres bound to 42543
2026-09-25 05:07:14.604 UTC [26248] FATAL: database "slave" does not exist
127.14.33.20:42543 - accepting connections
I20260925 05:07:14.604794 14468 mini_ranger.cc:161] Starting Ranger...
I20260925 05:07:14.619910 14468 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 05:07:15.046895 14468 mini_ranger.cc:87] Created ranger Postgres database
I20260925 05:07:15.046964 14468 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-admin
2026-09-25 05:07:15,303 [I] DB FLAVOR :POSTGRES
2026-09-25 05:07:15,303 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:07:15,303 [I] Checking connection..
2026-09-25 05:07:15,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 05:07:15,529 [I] Checking connection passed.
2026-09-25 05:07:15,530 [I] --------- Verifying version history table ---------
2026-09-25 05:07:15,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:07:15,763 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:07:15,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:07:16,005 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 05:07:16,005 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 05:07:16,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 05:07:16.253 UTC [26312] WARNING: there is no transaction in progress
2026-09-25 05:07:16,259 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 05:07:16,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 05:07:16,492 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 05:07:16,493 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:07:16,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 05:07:16,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 05:07:17,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4dmq.c.gcp-upstream.internal','N') ;"
2026-09-25 05:07:17,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:07:17,497 [I] Table x_portal_user does not exist in database ranger
2026-09-25 05:07:17,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:07:17,748 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 05:07:17,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:07:17,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:07:18,231 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 05:07:18,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 05:07:19.134 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.150 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.166 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.181 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.195 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.210 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.259 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.266 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.279 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.288 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.298 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.310 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.319 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.325 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.331 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19.337 UTC [26420] WARNING: there is no transaction in progress
2026-09-25 05:07:19,910 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 05:07:19,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 05:07:20,123 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 05:07:20,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 05:07:20,375 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 05:07:20,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 05:07:20,617 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 05:07:20,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 05:07:20,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 05:07:21,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4dmq.c.gcp-upstream.internal';"
2026-09-25 05:07:21,359 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 05:07:21,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 05:07:21,608 [I] DB_PATCHES have already been applied
I20260925 05:07:21.613834 14468 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-admin:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task85DeA_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 05:07:21.613909 14468 mini_ranger.cc:193] Using host: 127.14.33.20
I20260925 05:07:21.616976 14468 mini_ranger.cc:227] Ranger admin URL: http://127.14.33.20:50049
Sep 25, 2026 5:07:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:07:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 5:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 5:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 5:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 5:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 05:07:22.733453 14468 mini_ranger.cc:160] Time spent starting Ranger: real 8.129s user 0.002s sys 0.007s
Sep 25, 2026 5:07:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 5:07:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 5:07:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 5:07:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 5:07:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 05:07:43.729770 14468 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.33.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.33.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/rangerkms_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.33.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/rangerkms_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.33.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.33.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.33.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/HTTP_127.14.33.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 05:07:43 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312863, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 05:07:43.806602 14468 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 05:07:43.814723 14468 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 05:07:44.585534 14468 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 05:07:44.585632 14468 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms
I20260925 05:07:44.585657 14468 mini_ranger_kms.cc:227] Using postgres at 127.14.33.20:42543
2026-09-25 05:07:44,742 [I] DB FLAVOR :POSTGRES
2026-09-25 05:07:44,742 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 05:07:44,742 [I] Checking connection
2026-09-25 05:07:44,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 05:07:44,962 [I] connection success
2026-09-25 05:07:44,962 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 05:07:44,963 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 05:07:44,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 05:07:45,195 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 05:07:45,196 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 05:07:45,196 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 05:07:45,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.33.20:42543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 05:07:45,440 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 05:07:45.913223 14468 mini_ranger_kms.cc:310] Created kms service
I20260925 05:07:46.109092 14468 mini_ranger_kms.cc:326] Created kudu user
I20260925 05:07:46.149642 14468 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 05:07:46 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312863, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:46.544450 14468 mini_ranger_kms.cc:384] Added ranger policy
I20260925 05:07:46.544654 14468 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms:/tmp/dist-test-task85DeA_/build/release/bin/postgresql.jar:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task85DeA_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 05:07:46.544687 14468 mini_ranger_kms.cc:240] Using host: 127.14.33.20
I20260925 05:07:46.548085 14468 mini_ranger_kms.cc:276] Ranger KMS PID: 26600
I20260925 05:07:46.548166 14468 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.33.20:56701
Sep 25, 2026 5:07:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 5:07:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 5:07:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 5:07:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 5:07:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 5:07:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 05:07:48.734304 14468 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.928s user 0.000s sys 0.140s
I20260925 05:07:48.734400 14468 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 05:07:48.734445 14468 mini_ranger_kms.cc:401] 127.14.33.20:56701/kms/v1/keys
Sep 25 05:07:48 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.33.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 05:07:49 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.33.20@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
Sep 25 05:07:49 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312868, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.33.20@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
Sep 25 05:07:51 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:07:51.774060 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41341
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41341
--ranger_config_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:51.877676 26651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:51.877856 26651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:51.877875 26651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:51.878994 26651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:51.879035 26651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:51.879050 26651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:51.879061 26651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:51.879073 26651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:51.880509 26651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.33.20:56701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41341
--ranger_config_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:51.880728 26651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:51.880962 26651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:07:51.886207 26651 server_base.cc:1034] running on GCE node
W20260925 05:07:51.886174 26656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:51.886174 26657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:51.886509 26659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:51.886777 26651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:51.887064 26651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:51.888230 26651 hybrid_clock.cc:648] HybridClock initialized: now 1790312871888218 us; error 30 us; skew 500 ppm
Sep 25 05:07:51 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312871, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:51.898985 26651 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:07:51.899487 26651 webserver.cc:460] Webserver started at http://127.14.33.62:44309/ using document root <none> and password file <none>
I20260925 05:07:51.899700 26651 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:51.899757 26651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:51.899910 26651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:51 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312871, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:51.947932 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "acb5ea8ad0bf4ba2926bdb66ff504302"
format_stamp: "Formatted at 2026-09-25 05:07:51 on dist-test-slave-4dmq"
server_key: "fbb13e31caf3100c6c0cdc59692b2377"
server_key_iv: "b80c908f164752fb46fcba08babf17a4"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:51.948380 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "acb5ea8ad0bf4ba2926bdb66ff504302"
format_stamp: "Formatted at 2026-09-25 05:07:51 on dist-test-slave-4dmq"
server_key: "fbb13e31caf3100c6c0cdc59692b2377"
server_key_iv: "b80c908f164752fb46fcba08babf17a4"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:51.949966 26651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 05:07:51.963477 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:51.963701 26651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:51.963800 26651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "acb5ea8ad0bf4ba2926bdb66ff504302"
format_stamp: "Formatted at 2026-09-25 05:07:51 on dist-test-slave-4dmq"
server_key: "fbb13e31caf3100c6c0cdc59692b2377"
server_key_iv: "b80c908f164752fb46fcba08babf17a4"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:51.963865 26651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:51.976073 26651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:51.977998 26651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:51.978149 26651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:51.982656 26651 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:41341
I20260925 05:07:51.982700 26718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:41341 every 8 connection(s)
I20260925 05:07:51.983054 26651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:51.983711 26719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:51.985908 26719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Bootstrap starting.
I20260925 05:07:51.986459 26719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:51.986739 26719 log.cc:824] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:51.987421 26719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: No bootstrap required, opened a new log
I20260925 05:07:51.989399 26719 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:51.989535 26719 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:51.989557 26719 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb5ea8ad0bf4ba2926bdb66ff504302, State: Initialized, Role: FOLLOWER
I20260925 05:07:51.989723 26719 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:51.989785 26719 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:51.989858 26719 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:51.989912 26719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:51.989979 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 26651
I20260925 05:07:51.990100 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:07:51.990290 14468 external_mini_cluster.cc:1428] Setting key d19b141be0d93a264626f6734301095d
I20260925 05:07:51.990641 26719 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:51.990815 26719 leader_election.cc:302] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb5ea8ad0bf4ba2926bdb66ff504302; no voters:
I20260925 05:07:51.991003 26719 leader_election.cc:288] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:07:51.991027 26724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:51.991127 26724 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 LEADER]: Becoming Leader. State: Replica: acb5ea8ad0bf4ba2926bdb66ff504302, State: Running, Role: LEADER
I20260925 05:07:51.991225 26724 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:51.991251 26719 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:51.993239 26719 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:07:51.993564 26724 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } } }
I20260925 05:07:51.995059 26724 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:51.995394 26724 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb5ea8ad0bf4ba2926bdb66ff504302. Latest consensus state: current_term: 1 leader_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } } }
I20260925 05:07:51.995482 26724 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:51.996922 26741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:51.997450 26741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:07:51 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:52.003749 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:51.990747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.14.33.62:41341)
0925 05:07:51.990923 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:51.990933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:51.991453 (+ 520us) server_negotiation.cc:407] Connection header received
0925 05:07:51.991594 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:51.991601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:51.991672 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:51.991814 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:51.992621 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:51.996467 (+ 3846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:51.997556 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:51.997951 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:51.998082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:52.000409 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:52.000413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:52.000413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:52.000423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:52.002159 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:52.002553 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:52.002556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:52.002556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:52.002638 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:52.002853 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:52.002853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:52.002854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:52.003106 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:52.003283 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:52.003346 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 05:07:52.003467 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260925 05:07:52.009599 26741 catalog_manager.cc:1347] Generated new cluster ID: 418b24e4070f4ba18458cc2ab870754c
I20260925 05:07:52.009678 26741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:52.065793 26741 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:52.066675 26741 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:52.083815 26741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Generated new TSK 0
I20260925 05:07:52.084088 26741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:07:52.688474 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41341
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:52.806218 26765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:52.806486 26765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:52.806523 26765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:52.808148 26765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:52.808208 26765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:52.808246 26765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:07:52.810341 26765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.33.20:56701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:52.810572 26765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:52.810899 26765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:52.816227 26771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:52.816349 26773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:52.816237 26770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:52.816445 26765 server_base.cc:1034] running on GCE node
I20260925 05:07:52.816617 26765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:52.816859 26765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:52.818008 26765 hybrid_clock.cc:648] HybridClock initialized: now 1790312872817987 us; error 40 us; skew 500 ppm
Sep 25 05:07:52 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312872, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:52.820578 26765 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:07:52.821138 26765 webserver.cc:460] Webserver started at http://127.14.33.1:35677/ using document root <none> and password file <none>
I20260925 05:07:52.821424 26765 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:52.821502 26765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:52.821648 26765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:52 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312872, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:52.835659 26765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "55331adfcd43423aa12ca98420006437"
format_stamp: "Formatted at 2026-09-25 05:07:52 on dist-test-slave-4dmq"
server_key: "dbb43d29a6b9a067822060a58f9b7317"
server_key_iv: "3019208674890ea9be891a873641de4f"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:52.836179 26765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "55331adfcd43423aa12ca98420006437"
format_stamp: "Formatted at 2026-09-25 05:07:52 on dist-test-slave-4dmq"
server_key: "dbb43d29a6b9a067822060a58f9b7317"
server_key_iv: "3019208674890ea9be891a873641de4f"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:52.838126 26765 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 05:07:52.850767 26780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:52.851003 26765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:52.851099 26765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "55331adfcd43423aa12ca98420006437"
format_stamp: "Formatted at 2026-09-25 05:07:52 on dist-test-slave-4dmq"
server_key: "dbb43d29a6b9a067822060a58f9b7317"
server_key_iv: "3019208674890ea9be891a873641de4f"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:52.851181 26765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:52.874768 26765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:52.877388 26765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:52.877542 26765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:52.877769 26765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:52.878110 26765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:52.878180 26765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:52.878226 26765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:52.878274 26765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:52.885481 26765 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:44397
I20260925 05:07:52.885604 26895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:44397 every 8 connection(s)
I20260925 05:07:52.885905 26765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:07:52 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312872, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:52.894696 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 26765
I20260925 05:07:52.894809 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:07:52.895332 14468 external_mini_cluster.cc:1428] Setting key f19e17038c938a4da80a4a8fa5b1593d
I20260925 05:07:52.897759 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:52.886628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41341 (local address 127.14.33.1:51821)
0925 05:07:52.886819 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:52.886867 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:52.887496 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:52.887963 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:52.887968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:52.888201 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:52.888475 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:52.888481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:52.889490 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:52.889491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:52.890782 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:52.890785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:52.891332 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:52.891333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:52.891527 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:52.892126 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:52.892135 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:52.893841 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:52.896200 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:52.896206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:52.896209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:52.896499 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:52.896713 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:52.896714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:52.896714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:52.896831 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:52.897106 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:52.897111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:52.897305 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 05:07:52.897394 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:07:52.897535 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260925 05:07:52.898236 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:52.886871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:51821 (local address 127.14.33.62:41341)
0925 05:07:52.887701 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:52.887703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:52.887713 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:07:52.887740 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:52.887741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:52.887771 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:52.887852 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:52.888595 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:52.889393 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:52.890899 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:52.891208 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:52.891289 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:52.893941 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:52.893942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:52.893943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:52.893950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:52.896117 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:52.896585 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:52.896586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:52.896586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:52.896648 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:52.896904 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:52.896905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:52.896905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:52.897032 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:52.898019 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:52.898061 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 05:07:52.898141 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":66,"threads_started":1}
I20260925 05:07:52.898573 26896 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41341
I20260925 05:07:52.898685 26896 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:52.898877 26896 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:52.899395 26683 ts_manager.cc:194] Registered new tserver with Master: 55331adfcd43423aa12ca98420006437 (127.14.33.1:44397)
I20260925 05:07:52.900200 26683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:51821
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:07:52.925454 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41341
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:53.036767 26902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:53.036993 26902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:53.037030 26902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:53.038349 26902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:53.038415 26902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:53.038458 26902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:07:53.040096 26902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.33.20:56701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:53.040369 26902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:53.040606 26902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:53.045850 26912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.046046 26910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.046209 26909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.046063 26902 server_base.cc:1034] running on GCE node
I20260925 05:07:53.046478 26902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:53.046748 26902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:53.047899 26902 hybrid_clock.cc:648] HybridClock initialized: now 1790312873047879 us; error 36 us; skew 500 ppm
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.050626 26902 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:07:53.051051 26902 webserver.cc:460] Webserver started at http://127.14.33.2:46323/ using document root <none> and password file <none>
I20260925 05:07:53.051293 26902 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:53.051362 26902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:53.051493 26902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.063362 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "bcd5a66d103a4d26822302cb9bb8e666"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "2c7d64e300bcdad28c551d7f30ac8cf8"
server_key_iv: "cda231fe68bb0f252a7750f265a3199e"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.063814 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "bcd5a66d103a4d26822302cb9bb8e666"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "2c7d64e300bcdad28c551d7f30ac8cf8"
server_key_iv: "cda231fe68bb0f252a7750f265a3199e"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.069591 26902 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.001s
I20260925 05:07:53.097371 26919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.097674 26902 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.001s
I20260925 05:07:53.097807 26902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "bcd5a66d103a4d26822302cb9bb8e666"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "2c7d64e300bcdad28c551d7f30ac8cf8"
server_key_iv: "cda231fe68bb0f252a7750f265a3199e"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.097931 26902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:53.122224 26902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:53.124538 26902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:53.124696 26902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:53.124963 26902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:53.125413 26902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:53.125473 26902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.125504 26902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:53.125520 26902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.131942 26902 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:45299
I20260925 05:07:53.132424 26902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:53.133199 27033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:45299 every 8 connection(s)
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:53.141330 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 26902
I20260925 05:07:53.141435 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:07:53.141605 14468 external_mini_cluster.cc:1428] Setting key 06574ec92a96f0f8a67f37551a86a6d2
I20260925 05:07:53.144222 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.133851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:52401 (local address 127.14.33.62:41341)
0925 05:07:53.133903 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:53.133905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:53.134147 (+ 242us) server_negotiation.cc:407] Connection header received
0925 05:07:53.134791 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:53.134793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:53.134805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:53.134873 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:53.135563 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.136383 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.137908 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.138241 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.138320 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.140703 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:53.140706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:53.140706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:53.140715 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:53.142556 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.143065 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.143066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.143066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.143130 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.143427 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.143428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.143428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.143564 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:53.143611 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:53.144006 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 05:07:53.144106 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 05:07:53.144694 27036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.133817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41341 (local address 127.14.33.2:52401)
0925 05:07:53.133978 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:53.134024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:53.134678 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:53.134946 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:53.134951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:53.135204 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:53.135461 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:53.135467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.136540 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.136542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:53.137818 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:53.137821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.138343 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.138344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:53.138474 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.139025 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:53.139035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:53.140599 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:53.142651 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:53.142658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:53.142660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:53.142958 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:53.143217 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:53.143217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:53.143218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:53.143348 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:53.143663 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:53.143669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:53.143835 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 05:07:53.144308 (+ 473us) client_negotiation.cc:240] Negotiation successful
0925 05:07:53.144467 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260925 05:07:53.145323 27034 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41341
I20260925 05:07:53.145419 27034 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:53.145593 27034 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:53.146015 26683 ts_manager.cc:194] Registered new tserver with Master: bcd5a66d103a4d26822302cb9bb8e666 (127.14.33.2:45299)
I20260925 05:07:53.146519 26683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:52401
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:07:53.171864 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41341
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:53.286674 27039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:53.286926 27039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:53.286964 27039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:53.288091 27039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:53.288162 27039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:53.288210 27039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:07:53.289637 27039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.33.20:56701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:53.289891 27039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:53.290109 27039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:53.295128 27044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.295176 27045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.295408 27039 server_base.cc:1034] running on GCE node
W20260925 05:07:53.295166 27047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.295651 27039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:53.295907 27039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:53.297072 27039 hybrid_clock.cc:648] HybridClock initialized: now 1790312873297053 us; error 40 us; skew 500 ppm
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.299490 27039 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:07:53.299975 27039 webserver.cc:460] Webserver started at http://127.14.33.3:33817/ using document root <none> and password file <none>
I20260925 05:07:53.300207 27039 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:53.300249 27039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:53.300386 27039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:53.316623 27039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "c2caf777c523432c98bc0331bfe744e5"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "1b8612bcc911e0c2c66f258f1887c151"
server_key_iv: "dbfa350d1a528a3c8ba39af64aeb8da6"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.317083 27039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "c2caf777c523432c98bc0331bfe744e5"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "1b8612bcc911e0c2c66f258f1887c151"
server_key_iv: "dbfa350d1a528a3c8ba39af64aeb8da6"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.333580 27039 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.001s sys 0.000s
I20260925 05:07:53.361297 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.361598 27039 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.000s
I20260925 05:07:53.361723 27039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "c2caf777c523432c98bc0331bfe744e5"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "1b8612bcc911e0c2c66f258f1887c151"
server_key_iv: "dbfa350d1a528a3c8ba39af64aeb8da6"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.361824 27039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:53.396703 27039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:53.399171 27039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:53.399401 27039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:53.399699 27039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:53.400111 27039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:53.400213 27039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.400274 27039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:53.400324 27039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.407330 27039 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:33821
I20260925 05:07:53.407790 27039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:53.408520 27167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:33821 every 8 connection(s)
I20260925 05:07:53.410529 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27039
I20260925 05:07:53.410629 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:07:53.410822 14468 external_mini_cluster.cc:1428] Setting key 31ac3896e33bcae8ec450fa532adeb7b
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:53.420475 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.410078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:42219 (local address 127.14.33.62:41341)
0925 05:07:53.410119 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:53.410121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:53.410132 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:07:53.410725 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:53.410727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:53.410739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:53.410768 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:53.411548 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.412311 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.413741 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.414046 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.414121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.417198 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:53.417200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:53.417201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:53.417208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:53.418920 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.419392 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.419393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.419393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.419449 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.419724 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.419725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.419734 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.419888 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:53.419968 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:53.420297 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 05:07:53.420380 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:07:53.420612 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.409769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41341 (local address 127.14.33.3:42219)
0925 05:07:53.409948 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:53.409992 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:53.410592 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:53.410945 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:53.410950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:53.411182 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:53.411448 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:53.411455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.412417 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.412418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:53.413639 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:53.413642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.414144 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.414145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:53.414277 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.414830 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:53.414847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:53.416349 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:53.419021 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:53.419027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:53.419030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:53.419307 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:53.419516 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:53.419517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:53.419517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:53.419631 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:53.419967 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:53.419972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:53.420149 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 05:07:53.420259 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:07:53.420398 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260925 05:07:53.421458 27168 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41341
I20260925 05:07:53.421552 27168 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:53.421732 27168 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:53.422104 26683 ts_manager.cc:194] Registered new tserver with Master: c2caf777c523432c98bc0331bfe744e5 (127.14.33.3:33821)
I20260925 05:07:53.422602 26683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:42219
I20260925 05:07:53.423393 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:53.424059 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 26765
I20260925 05:07:53.431326 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 26902
I20260925 05:07:53.437682 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27039
I20260925 05:07:53.443050 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 26651
I20260925 05:07:53.449669 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41341
--webserver_interface=127.14.33.62
--webserver_port=44309
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41341
--ranger_config_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:53.526845 27171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:53.527027 27171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:53.527047 27171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:53.528177 27171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:53.528218 27171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:53.528275 27171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:53.528301 27171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:53.528331 27171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:53.529975 27171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41341
--ranger_config_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=44309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:53.530195 27171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:53.530396 27171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:53.534803 27176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.534889 27171 server_base.cc:1034] running on GCE node
W20260925 05:07:53.534811 27177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.534811 27179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.535315 27171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:53.535586 27171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:53.536741 27171 hybrid_clock.cc:648] HybridClock initialized: now 1790312873536724 us; error 29 us; skew 500 ppm
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.539009 27171 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:07:53.539402 27171 webserver.cc:460] Webserver started at http://127.14.33.62:44309/ using document root <none> and password file <none>
I20260925 05:07:53.539605 27171 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:53.539677 27171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:53.599639 27171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:53.600607 27187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.600839 27171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:53.600934 27171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "acb5ea8ad0bf4ba2926bdb66ff504302"
format_stamp: "Formatted at 2026-09-25 05:07:51 on dist-test-slave-4dmq"
server_key: "fbb13e31caf3100c6c0cdc59692b2377"
server_key_iv: "b80c908f164752fb46fcba08babf17a4"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.601429 27171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:53.609251 27171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:53.610741 27171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:53.610893 27171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:53.614905 27171 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:41341
I20260925 05:07:53.615010 27239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:41341 every 8 connection(s)
I20260925 05:07:53.615351 27171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:53.616544 27240 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:07:53.617218 27240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Bootstrap starting.
I20260925 05:07:53.625166 27240 log.cc:824] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:53.625231 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27171
I20260925 05:07:53.625524 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:44397
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:41341
--webserver_port=35677
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:53.626746 27240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:07:53.627015 27240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Bootstrap complete.
I20260925 05:07:53.630666 27240 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:53.631059 27240 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb5ea8ad0bf4ba2926bdb66ff504302, State: Initialized, Role: FOLLOWER
I20260925 05:07:53.631230 27240 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:53.631309 27240 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:07:53.631363 27240 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:07:53.631420 27240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:07:53.632874 27240 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:53.632985 27240 leader_election.cc:302] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb5ea8ad0bf4ba2926bdb66ff504302; no voters:
I20260925 05:07:53.633157 27240 leader_election.cc:288] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:07:53.633221 27244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:07:53.633397 27244 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [term 2 LEADER]: Becoming Leader. State: Replica: acb5ea8ad0bf4ba2926bdb66ff504302, State: Running, Role: LEADER
I20260925 05:07:53.633409 27240 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:53.633579 27244 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } }
I20260925 05:07:53.633903 27245 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } } }
I20260925 05:07:53.633978 27245 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:53.634191 27246 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb5ea8ad0bf4ba2926bdb66ff504302. Latest consensus state: current_term: 2 leader_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb5ea8ad0bf4ba2926bdb66ff504302" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41341 } } }
I20260925 05:07:53.634313 27246 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:53.634433 27250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:53.635002 27250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:53.635190 27250 catalog_manager.cc:1259] Loaded cluster ID: 418b24e4070f4ba18458cc2ab870754c
I20260925 05:07:53.635264 27250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:53.635370 27240 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 05:07:53.638975 27250 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:53.639537 27250 catalog_manager.cc:5879] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Loaded TSK: 0
I20260925 05:07:53.639854 27250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:07:53.740455 27243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:53.740739 27243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:53.740794 27243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:53.742588 27243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:53.742645 27243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:53.742681 27243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:07:53.744673 27243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:44397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=35677
--tserver_master_addrs=127.14.33.62:41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:53.744899 27243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:53.745178 27243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:53.750980 27278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.751291 27280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.751394 27277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.751590 27243 server_base.cc:1034] running on GCE node
I20260925 05:07:53.751804 27243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:53.752076 27243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:53.753190 27243 hybrid_clock.cc:648] HybridClock initialized: now 1790312873753170 us; error 33 us; skew 500 ppm
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.756155 27243 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:07:53.756731 27243 webserver.cc:460] Webserver started at http://127.14.33.1:35677/ using document root <none> and password file <none>
I20260925 05:07:53.756963 27243 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:53.757025 27243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:53.781492 27243 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 05:07:53.782186 27288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.782390 27243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:53.782483 27243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "55331adfcd43423aa12ca98420006437"
format_stamp: "Formatted at 2026-09-25 05:07:52 on dist-test-slave-4dmq"
server_key: "dbb43d29a6b9a067822060a58f9b7317"
server_key_iv: "3019208674890ea9be891a873641de4f"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.782932 27243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:53.800184 27243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:53.802114 27243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:53.802290 27243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:53.802522 27243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:53.802879 27243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:53.802947 27243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.802987 27243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:53.803027 27243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.809742 27243 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:44397
I20260925 05:07:53.810207 27243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:07:53.810665 27401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:44397 every 8 connection(s)
I20260925 05:07:53.811553 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27243
I20260925 05:07:53.811825 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:45299
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:41341
--webserver_port=46323
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:53.827323 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.814637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41341 (local address 127.14.33.1:59925)
0925 05:07:53.814855 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:53.814908 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:53.815634 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:53.816361 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:53.816366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:53.816637 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:53.816955 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:53.816961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.818836 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.818837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:53.820165 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:53.820168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.821125 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.821126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:53.821298 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.821878 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:53.821888 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:53.823492 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:53.825669 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:53.825675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:53.825677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:53.825932 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:53.826180 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:53.826181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:53.826181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:53.826298 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:53.826702 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:53.826706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:53.826864 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 05:07:53.826983 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 05:07:53.827108 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260925 05:07:53.828291 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:53.815034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:59925 (local address 127.14.33.62:41341)
0925 05:07:53.815859 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:53.815867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:53.815908 (+ 41us) server_negotiation.cc:407] Connection header received
0925 05:07:53.816063 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:53.816069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:53.816128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:53.817086 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:53.817203 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.818701 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.820295 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:53.820988 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:53.821562 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:53.823615 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:53.823638 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:53.823638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:53.823649 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:53.825578 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.826017 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.826020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.826020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.826112 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:53.826366 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:53.826366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:53.826367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:53.826632 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:53.827893 (+ 1261us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:53.827984 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:07:53.828095 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":81,"threads_started":1}
I20260925 05:07:53.829170 27402 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41341
I20260925 05:07:53.829264 27402 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:53.829444 27402 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
W20260925 05:07:53.830204 27402 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:41341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:07:53.830271 27402 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:53.830322 27402 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:53.830672 27402 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:53.830729 27402 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
W20260925 05:07:53.830996 27402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:07:53.911188 27404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:53.911556 27404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:53.911648 27404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:53.913378 27404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:53.913479 27404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:53.913559 27404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:07:53.915624 27404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:45299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=46323
--tserver_master_addrs=127.14.33.62:41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:53.916061 27404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:53.916363 27404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:53.922437 27412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.922794 27414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:53.925287 27411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:53.926046 27404 server_base.cc:1034] running on GCE node
I20260925 05:07:53.926383 27404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:53.926674 27404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:53.929925 27404 hybrid_clock.cc:648] HybridClock initialized: now 1790312873929895 us; error 49 us; skew 500 ppm
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:53.933338 27404 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:07:53.933990 27404 webserver.cc:460] Webserver started at http://127.14.33.2:46323/ using document root <none> and password file <none>
I20260925 05:07:53.934319 27404 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:53.934427 27404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:07:53 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:53.949807 27404 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:53.950552 27422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.950791 27404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:53.950898 27404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "bcd5a66d103a4d26822302cb9bb8e666"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "2c7d64e300bcdad28c551d7f30ac8cf8"
server_key_iv: "cda231fe68bb0f252a7750f265a3199e"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:53.951453 27404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:53.993520 27404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:53.995656 27404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:53.995823 27404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:53.996104 27404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:53.996502 27404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:53.996554 27404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:53.996587 27404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:53.996618 27404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.003803 27404 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:45299
I20260925 05:07:54.004032 27535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:45299 every 8 connection(s)
I20260925 05:07:54.004329 27404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:07:54.007898 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27404
I20260925 05:07:54.008174 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.33.20:56701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:33821
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:41341
--webserver_port=33817
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:07:54 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312873, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:54.017203 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.005076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:36701 (local address 127.14.33.62:41341)
0925 05:07:54.005134 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:54.005136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:54.005475 (+ 339us) server_negotiation.cc:407] Connection header received
0925 05:07:54.006029 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:54.006031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:54.006042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:54.006118 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:54.006740 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.007246 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.008660 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.008966 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.009020 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.013582 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:54.013583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:54.013583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:54.013591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:54.015416 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.015930 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.015931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.015931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.015997 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.016344 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.016346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.016346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.016497 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:54.016546 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:54.016960 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 05:07:54.017066 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:07:54.017437 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.005086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41341 (local address 127.14.33.2:36701)
0925 05:07:54.005325 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:54.005369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:54.005917 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:54.006170 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:54.006173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:54.006408 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:54.006648 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.006655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.007346 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.007347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.008560 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.008563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.009115 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.009116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.009277 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.009797 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:54.009808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:54.013477 (+ 3669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:54.015509 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.015516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.015519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.015838 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.016077 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.016078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.016078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.016235 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.016597 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:54.016603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:54.016789 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 05:07:54.016913 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 05:07:54.017076 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":127,"threads_started":1}
I20260925 05:07:54.018050 27536 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41341
I20260925 05:07:54.018151 27536 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.018352 27536 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
W20260925 05:07:54.019023 27536 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:41341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:07:54.019102 27536 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.019177 27536 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:54.019544 27536 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.019613 27536 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
W20260925 05:07:54.019891 27536 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:07:54.102746 27539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:54.102950 27539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:54.102985 27539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:54.104279 27539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:54.104336 27539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:54.104420 27539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:07:54.106298 27539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.33.20:56701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=33817
--tserver_master_addrs=127.14.33.62:41341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:54.106714 27539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:54.107069 27539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:54.117894 27546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:54.118206 27545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:54.118690 27539 server_base.cc:1034] running on GCE node
W20260925 05:07:54.118808 27548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:54.119061 27539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:54.119369 27539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:54.120589 27539 hybrid_clock.cc:648] HybridClock initialized: now 1790312874120556 us; error 47 us; skew 500 ppm
Sep 25 05:07:54 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:54.123417 27539 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:07:54.123900 27539 webserver.cc:460] Webserver started at http://127.14.33.3:33817/ using document root <none> and password file <none>
I20260925 05:07:54.124118 27539 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:54.124169 27539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 05:07:54 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:54.147358 27539 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 05:07:54.148084 27556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.148269 27539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:07:54.148352 27539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "c2caf777c523432c98bc0331bfe744e5"
format_stamp: "Formatted at 2026-09-25 05:07:53 on dist-test-slave-4dmq"
server_key: "1b8612bcc911e0c2c66f258f1887c151"
server_key_iv: "dbfa350d1a528a3c8ba39af64aeb8da6"
server_key_version: "kuduclusterkey@0"
I20260925 05:07:54.148823 27539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:54.173808 27539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:54.176163 27539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:54.176311 27539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:54.176584 27539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:07:54.176961 27539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:07:54.177014 27539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.177044 27539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:07:54.177062 27539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:54.184343 27539 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:33821
I20260925 05:07:54.184931 27539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:07:54.184770 27669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:33821 every 8 connection(s)
I20260925 05:07:54.195510 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27539
Sep 25 05:07:54 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:54.200078 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.188738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:58083 (local address 127.14.33.62:41341)
0925 05:07:54.188789 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:54.188791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:54.189150 (+ 359us) server_negotiation.cc:407] Connection header received
0925 05:07:54.189946 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:54.189948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:54.189961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:54.189994 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:54.190797 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.191530 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.193020 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.193383 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.193465 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.196628 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:54.196629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:54.196630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:54.196637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:54.198388 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.198923 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.198925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.198925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.198985 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.199280 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.199281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.199281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.199421 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:54.199469 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:54.199897 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 05:07:54.199983 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:07:54.200228 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.188758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41341 (local address 127.14.33.3:58083)
0925 05:07:54.188934 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:54.189006 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:54.189818 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:54.190218 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:54.190223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:54.190438 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:54.190684 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.190691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.191645 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.191646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.192908 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:54.192911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.193547 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.193548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:54.193689 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.194301 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:54.194311 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:54.196513 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:54.198486 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.198492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.198494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.198821 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.199059 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:54.199059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:54.199060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:54.199185 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:54.199523 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:54.199529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:54.199702 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 05:07:54.199838 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 05:07:54.200001 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260925 05:07:54.200794 27670 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41341
I20260925 05:07:54.200889 27670 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.201085 27670 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
W20260925 05:07:54.201941 27670 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:41341 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 05:07:54.202020 27670 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.202086 27670 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:54.202445 27670 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.202522 27670 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
W20260925 05:07:54.202867 27670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:07:54.206295 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:54.195959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.14.33.62:41341)
0925 05:07:54.196223 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:54.196224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:54.196230 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:07:54.196246 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:54.196247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:54.196255 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:54.196326 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:54.197088 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.197855 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.200450 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:54.200767 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:54.200900 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:54.203259 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:54.203261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:54.203261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:54.203268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:54.205031 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.205462 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.205464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.205464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.205523 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:54.205764 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:54.205765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:54.205765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:54.205895 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:54.206073 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:54.206125 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:07:54.206203 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":62,"threads_started":1}
Sep 25 05:07:54 dist-test-slave-4dmq krb5kdc[26207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:54.831190 27402 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:54.831362 27402 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:54.831739 27204 ts_manager.cc:194] Registered new tserver with Master: 55331adfcd43423aa12ca98420006437 (127.14.33.1:44397)
I20260925 05:07:54.832630 27204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:59925
I20260925 05:07:55.020043 27536 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:55.020182 27536 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:55.020529 27204 ts_manager.cc:194] Registered new tserver with Master: bcd5a66d103a4d26822302cb9bb8e666 (127.14.33.2:45299)
I20260925 05:07:55.021021 27204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:36701
Sep 25 05:07:55 dist-test-slave-4dmq krb5kdc[26207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for HTTP/127.14.33.20@KRBTEST.COM
I20260925 05:07:55.203020 27670 heartbeater.cc:464] Registering TS with master...
I20260925 05:07:55.203172 27670 heartbeater.cc:511] Master 127.14.33.62:41341 requested a full tablet report, sending...
I20260925 05:07:55.203480 27204 ts_manager.cc:194] Registered new tserver with Master: c2caf777c523432c98bc0331bfe744e5 (127.14.33.3:33821)
I20260925 05:07:55.203994 27204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:58083
I20260925 05:07:55.205470 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:07:55.214025 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.206434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.14.33.62:41341)
0925 05:07:55.206667 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.206669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.206766 (+ 97us) server_negotiation.cc:407] Connection header received
0925 05:07:55.206854 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.206855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.206867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.206907 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:55.207644 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.208383 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.209401 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.209673 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.209710 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.211140 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:55.211142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:55.211142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:55.211149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:55.212807 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.213228 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.213229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.213229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.213291 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:55.213482 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:55.213482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:55.213483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:55.213590 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:55.213655 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.213849 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.213932 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":60,"threads_started":1}
I20260925 05:07:55.216368 27204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:07:55.218715 27204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:07:55.234068 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.227067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:45299 (local address 127.0.0.1:42278)
0925 05:07:55.229236 (+ 2169us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.229276 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:55.229337 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:55.230008 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:55.230011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:55.230013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:55.230069 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.230070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.231286 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.231287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.232776 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.232780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.233799 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.233800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.233873 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.233885 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:07:55.233947 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 05:07:55.233992 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2139,"thread_start_us":63,"threads_started":1}
I20260925 05:07:55.234892 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.227169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.14.33.2:45299)
0925 05:07:55.229802 (+ 2633us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.229807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.229817 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:07:55.229850 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.229852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.229867 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.229934 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:55.230149 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.231192 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.232872 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.233605 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.233693 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.233766 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.234781 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.234821 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2599,"thread_start_us":69,"threads_started":1}
I20260925 05:07:55.235666 27470 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba6d7f02a614ae8a35a6814636e1924 (DEFAULT_TABLE table=test-table [id=6cbc75773c5b402f953c60d68f08f51f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:55.236027 27470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba6d7f02a614ae8a35a6814636e1924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:55.237351 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.227254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:33821 (local address 127.0.0.1:37558)
0925 05:07:55.229430 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.229437 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:55.229468 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:55.233842 (+ 4374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:55.233843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:55.233845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:55.233890 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.233894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.235190 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.235191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.236206 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.236209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.237095 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.237096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.237189 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.237194 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:07:55.237249 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:07:55.237283 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"thread_start_us":57,"threads_started":1}
I20260925 05:07:55.237766 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.227423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.14.33.3:33821)
0925 05:07:55.233590 (+ 6167us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.233595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.233607 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:07:55.233641 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.233643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.233659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.233751 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:55.233976 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.235079 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.236325 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.236991 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.237083 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.237173 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.237644 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.237698 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6132,"thread_start_us":42,"threads_started":1}
I20260925 05:07:55.238709 27604 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba6d7f02a614ae8a35a6814636e1924 (DEFAULT_TABLE table=test-table [id=6cbc75773c5b402f953c60d68f08f51f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:55.239081 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba6d7f02a614ae8a35a6814636e1924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:55.243155 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.227586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44397 (local address 127.0.0.1:49476)
0925 05:07:55.234420 (+ 6834us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.234430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:55.234461 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:55.239934 (+ 5473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:55.239935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:55.239936 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:55.239963 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.239964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.241078 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.241079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.242067 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.242069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.242964 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.242965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.243016 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.243018 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:07:55.243060 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 05:07:55.243094 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6824,"thread_start_us":55,"threads_started":1}
I20260925 05:07:55.243448 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.227686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.14.33.1:44397)
0925 05:07:55.239717 (+ 12031us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.239734 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.239744 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:07:55.239774 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.239776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.239791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.239861 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:55.240026 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.240986 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.242166 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.242818 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.242885 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.242934 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.243341 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.243380 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11994,"thread_start_us":71,"threads_started":1}
I20260925 05:07:55.244052 27317 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba6d7f02a614ae8a35a6814636e1924 (DEFAULT_TABLE table=test-table [id=6cbc75773c5b402f953c60d68f08f51f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:07:55.244347 27317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba6d7f02a614ae8a35a6814636e1924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:55.245253 27703 tablet_bootstrap.cc:492] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Bootstrap starting.
I20260925 05:07:55.245946 27703 tablet_bootstrap.cc:654] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:55.246294 27703 log.cc:824] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:55.249560 27704 tablet_bootstrap.cc:492] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Bootstrap starting.
I20260925 05:07:55.249796 27703 tablet_bootstrap.cc:492] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: No bootstrap required, opened a new log
I20260925 05:07:55.249938 27703 ts_tablet_manager.cc:1397] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 05:07:55.250222 27704 tablet_bootstrap.cc:654] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:55.250553 27704 log.cc:824] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:55.250912 27706 tablet_bootstrap.cc:492] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437: Bootstrap starting.
I20260925 05:07:55.251515 27706 tablet_bootstrap.cc:654] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:55.251835 27706 log.cc:824] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:55.252290 27703 raft_consensus.cc:348] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.252457 27703 raft_consensus.cc:374] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:55.252497 27703 raft_consensus.cc:729] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd5a66d103a4d26822302cb9bb8e666, State: Initialized, Role: FOLLOWER
I20260925 05:07:55.252664 27703 consensus_queue.cc:260] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.252929 27703 ts_tablet_manager.cc:1428] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:55.253084 27536 heartbeater.cc:503] Master 127.14.33.62:41341 was elected leader, sending a full tablet report...
W20260925 05:07:55.255661 27537 tablet.cc:2369] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:55.258112 27704 tablet_bootstrap.cc:492] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: No bootstrap required, opened a new log
I20260925 05:07:55.258177 27704 ts_tablet_manager.cc:1397] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 05:07:55.260738 27704 raft_consensus.cc:348] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.260902 27704 raft_consensus.cc:374] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:55.260936 27704 raft_consensus.cc:729] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2caf777c523432c98bc0331bfe744e5, State: Initialized, Role: FOLLOWER
I20260925 05:07:55.261196 27704 consensus_queue.cc:260] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.261523 27704 ts_tablet_manager.cc:1428] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:55.261963 27670 heartbeater.cc:503] Master 127.14.33.62:41341 was elected leader, sending a full tablet report...
I20260925 05:07:55.263128 27706 tablet_bootstrap.cc:492] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437: No bootstrap required, opened a new log
I20260925 05:07:55.263252 27706 ts_tablet_manager.cc:1397] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260925 05:07:55.265565 27706 raft_consensus.cc:348] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.265910 27706 raft_consensus.cc:374] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:55.266002 27706 raft_consensus.cc:729] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55331adfcd43423aa12ca98420006437, State: Initialized, Role: FOLLOWER
I20260925 05:07:55.266165 27706 consensus_queue.cc:260] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.266469 27706 ts_tablet_manager.cc:1428] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:07:55.266605 27402 heartbeater.cc:503] Master 127.14.33.62:41341 was elected leader, sending a full tablet report...
I20260925 05:07:55.269248 27710 raft_consensus.cc:482] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:55.269337 27710 raft_consensus.cc:504] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.270015 27710 leader_election.cc:288] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55331adfcd43423aa12ca98420006437 (127.14.33.1:44397), bcd5a66d103a4d26822302cb9bb8e666 (127.14.33.2:45299)
I20260925 05:07:55.275660 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.269937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:39613 (local address 127.14.33.1:44397)
0925 05:07:55.269982 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.269984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.270190 (+ 206us) server_negotiation.cc:407] Connection header received
0925 05:07:55.270280 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.270280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.270292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.270342 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:55.272760 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.273467 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.274827 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.275331 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.275371 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.275381 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.275565 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.275600 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:07:55.275909 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.269762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44397 (local address 127.14.33.3:39613)
0925 05:07:55.270124 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.270136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:55.270206 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:55.272554 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:55.272555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:55.272557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:55.272656 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.272657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.273557 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.273557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.274689 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.274692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.275433 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.275434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.275495 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.275498 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:07:55.275786 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 05:07:55.275827 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":82,"threads_started":1}
I20260925 05:07:55.276180 27337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba6d7f02a614ae8a35a6814636e1924" candidate_uuid: "c2caf777c523432c98bc0331bfe744e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55331adfcd43423aa12ca98420006437" is_pre_election: true
I20260925 05:07:55.276353 27337 raft_consensus.cc:2454] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2caf777c523432c98bc0331bfe744e5 in term 0.
I20260925 05:07:55.276587 27558 leader_election.cc:302] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55331adfcd43423aa12ca98420006437, c2caf777c523432c98bc0331bfe744e5; no voters:
I20260925 05:07:55.277238 27712 raft_consensus.cc:2781] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:55.277315 27712 raft_consensus.cc:482] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:55.277338 27712 raft_consensus.cc:3037] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:55.278131 27712 raft_consensus.cc:504] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.278390 27337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba6d7f02a614ae8a35a6814636e1924" candidate_uuid: "c2caf777c523432c98bc0331bfe744e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55331adfcd43423aa12ca98420006437"
I20260925 05:07:55.278460 27337 raft_consensus.cc:3037] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:55.279229 27337 raft_consensus.cc:2454] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2caf777c523432c98bc0331bfe744e5 in term 1.
I20260925 05:07:55.279387 27558 leader_election.cc:302] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55331adfcd43423aa12ca98420006437, c2caf777c523432c98bc0331bfe744e5; no voters:
I20260925 05:07:55.279465 27710 raft_consensus.cc:2781] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:55.279603 27710 raft_consensus.cc:686] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 1 LEADER]: Becoming Leader. State: Replica: c2caf777c523432c98bc0331bfe744e5, State: Running, Role: LEADER
I20260925 05:07:55.279691 27710 consensus_queue.cc:237] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } }
I20260925 05:07:55.280416 27204 catalog_manager.cc:5495] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2caf777c523432c98bc0331bfe744e5 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "c2caf777c523432c98bc0331bfe744e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2caf777c523432c98bc0331bfe744e5" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 33821 } health_report { overall_health: HEALTHY } } }
I20260925 05:07:55.280616 27712 leader_election.cc:288] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [CANDIDATE]: Term 1 election: Requested vote from peers 55331adfcd43423aa12ca98420006437 (127.14.33.1:44397), bcd5a66d103a4d26822302cb9bb8e666 (127.14.33.2:45299)
I20260925 05:07:55.282125 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.272776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:57905 (local address 127.14.33.2:45299)
0925 05:07:55.272816 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.272817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.276151 (+ 3334us) server_negotiation.cc:407] Connection header received
0925 05:07:55.276252 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.276254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.276267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.276298 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:07:55.276468 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.277154 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.281045 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.281605 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.281646 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.281656 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.282023 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.282063 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:07:55.282321 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.272597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:45299 (local address 127.14.33.3:57905)
0925 05:07:55.276101 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.276111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:55.276178 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:55.276360 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:55.276361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:55.276362 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:07:55.276392 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.276393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.279500 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.279501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.280498 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:55.280500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.281897 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.281898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:55.281952 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.281955 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:07:55.282230 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 05:07:55.282267 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3493,"thread_start_us":75,"threads_started":1}
I20260925 05:07:55.282612 27490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba6d7f02a614ae8a35a6814636e1924" candidate_uuid: "c2caf777c523432c98bc0331bfe744e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd5a66d103a4d26822302cb9bb8e666" is_pre_election: true
I20260925 05:07:55.282754 27490 raft_consensus.cc:2454] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2caf777c523432c98bc0331bfe744e5 in term 0.
I20260925 05:07:55.282857 27489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba6d7f02a614ae8a35a6814636e1924" candidate_uuid: "c2caf777c523432c98bc0331bfe744e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcd5a66d103a4d26822302cb9bb8e666"
I20260925 05:07:55.282907 27489 raft_consensus.cc:3037] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:55.283636 27489 raft_consensus.cc:2454] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2caf777c523432c98bc0331bfe744e5 in term 1.
I20260925 05:07:55.289705 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.286659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.14.33.3:33821)
0925 05:07:55.286693 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:55.286695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:55.286702 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:07:55.286773 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:55.286774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:55.286790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:55.286851 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:07:55.287034 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.287687 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.288698 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:55.288959 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:55.289007 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:55.289216 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:07:55.289326 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:07:55.289392 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:55.289614 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 05:07:55.289632 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:07:55.291512 27489 raft_consensus.cc:1261] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 1 FOLLOWER]: Refusing update from remote peer c2caf777c523432c98bc0331bfe744e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:55.291798 27715 consensus_queue.cc:1035] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd5a66d103a4d26822302cb9bb8e666" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:55.293658 27337 raft_consensus.cc:1261] T 2ba6d7f02a614ae8a35a6814636e1924 P 55331adfcd43423aa12ca98420006437 [term 1 FOLLOWER]: Refusing update from remote peer c2caf777c523432c98bc0331bfe744e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:55.293926 27715 consensus_queue.cc:1035] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55331adfcd43423aa12ca98420006437" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:55.297528 27718 mvcc.cc:205] Tried to move back new op lower bound from 7333121537191641088 to 7333121537146294272. Current Snapshot: MvccSnapshot[applied={T|T < 7333121537191641088}]
I20260925 05:07:55.297978 27721 mvcc.cc:205] Tried to move back new op lower bound from 7333121537191641088 to 7333121537146294272. Current Snapshot: MvccSnapshot[applied={T|T < 7333121537191641088}]
I20260925 05:07:55.300052 27719 mvcc.cc:205] Tried to move back new op lower bound from 7333121537191641088 to 7333121537146294272. Current Snapshot: MvccSnapshot[applied={T|T < 7333121537191641088}]
I20260925 05:07:55.302192 27203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:55.302285 27203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:07:55.302948 27203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acb5ea8ad0bf4ba2926bdb66ff504302: Sending DeleteTablet for 3 replicas of tablet 2ba6d7f02a614ae8a35a6814636e1924
I20260925 05:07:55.303534 27317 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba6d7f02a614ae8a35a6814636e1924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:55 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260925 05:07:55.303812 27470 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba6d7f02a614ae8a35a6814636e1924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:55 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:42278
I20260925 05:07:55.304028 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27243
I20260925 05:07:55.304029 27604 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba6d7f02a614ae8a35a6814636e1924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:07:55 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:37558
I20260925 05:07:55.313206 27729 tablet_replica.cc:330] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: stopping tablet replica
I20260925 05:07:55.313320 27729 raft_consensus.cc:2229] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:07:55.313441 27729 raft_consensus.cc:2258] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:55.313884 27729 ts_tablet_manager.cc:1905] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:55.318686 27729 ts_tablet_manager.cc:1918] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:55.318773 27729 log.cc:1196] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/2ba6d7f02a614ae8a35a6814636e1924
I20260925 05:07:55.319022 27729 ts_tablet_manager.cc:1939] T 2ba6d7f02a614ae8a35a6814636e1924 P c2caf777c523432c98bc0331bfe744e5: Deleting consensus metadata
I20260925 05:07:55.319413 27188 catalog_manager.cc:4841] TS c2caf777c523432c98bc0331bfe744e5 (127.14.33.3:33821): tablet 2ba6d7f02a614ae8a35a6814636e1924 (table test-table [id=6cbc75773c5b402f953c60d68f08f51f]) successfully deleted
I20260925 05:07:55.319602 27728 tablet_replica.cc:330] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: stopping tablet replica
I20260925 05:07:55.319763 27728 raft_consensus.cc:2229] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:07:55.319877 27728 raft_consensus.cc:2258] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:07:55.320297 27728 ts_tablet_manager.cc:1905] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:07:55.321806 27728 ts_tablet_manager.cc:1918] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:07:55.321882 27728 log.cc:1196] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/2ba6d7f02a614ae8a35a6814636e1924
I20260925 05:07:55.322145 27728 ts_tablet_manager.cc:1939] T 2ba6d7f02a614ae8a35a6814636e1924 P bcd5a66d103a4d26822302cb9bb8e666: Deleting consensus metadata
I20260925 05:07:55.322533 27188 catalog_manager.cc:4841] TS bcd5a66d103a4d26822302cb9bb8e666 (127.14.33.2:45299): tablet 2ba6d7f02a614ae8a35a6814636e1924 (table test-table [id=6cbc75773c5b402f953c60d68f08f51f]) successfully deleted
W20260925 05:07:55.322808 27189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:44397 (error 108)
I20260925 05:07:55.323386 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27404
I20260925 05:07:55.337549 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:55.337394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44397 (local address 127.0.0.1:49482)
0925 05:07:55.337459 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:55.337502 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 05:07:55.337684 27189 catalog_manager.cc:4584] TS 55331adfcd43423aa12ca98420006437 (127.14.33.1:44397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ba6d7f02a614ae8a35a6814636e1924: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44397: connect: Connection refused (error 111)
I20260925 05:07:55.341377 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27539
I20260925 05:07:55.347215 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27171
I20260925 05:07:55.352269 14468 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 05:07:55.696425 14468 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 05:07:55.696542 14468 mini_ranger.cc:66] Stopping Ranger...
I20260925 05:07:56.040503 14468 mini_ranger.cc:68] Stopped Ranger
2026-09-25 05:07:56.040 UTC [26237] LOG: received smart shutdown request
2026-09-25 05:07:56.044 UTC [26237] LOG: background worker "logical replication launcher" (PID 26246) exited with exit code 1
2026-09-25 05:07:56.044 UTC [26241] LOG: shutting down
2026-09-25 05:07:56.189 UTC [26237] LOG: database system is shut down
2026-09-25T05:07:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46128 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:56 dist-test-slave-4dmq krb5kdc[27739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:56 dist-test-slave-4dmq krb5kdc[27739](info): set up 2 sockets
Sep 25 05:07:56 dist-test-slave-4dmq krb5kdc[27739](info): commencing operation
krb5kdc: starting...
W20260925 05:07:57.302076 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:07:57.408649 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:45369
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:45369,127.14.33.61:45101,127.14.33.60:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:07:57.481478 27752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:57.481667 27752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:57.481688 27752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:57.482856 27752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:57.482899 27752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:57.482913 27752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:57.482946 27752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:57.482977 27752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:57.484387 27752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:45369,127.14.33.61:45101,127.14.33.60:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:57.484627 27752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:57.484838 27752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:57.489260 27758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.489253 27757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.489312 27752 server_base.cc:1034] running on GCE node
W20260925 05:07:57.489274 27760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.489763 27752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:57.490027 27752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:57.491166 27752 hybrid_clock.cc:648] HybridClock initialized: now 1790312877491153 us; error 29 us; skew 500 ppm
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:57.493641 27752 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:07:57.494024 27752 webserver.cc:460] Webserver started at http://127.14.33.62:37573/ using document root <none> and password file <none>
I20260925 05:07:57.494242 27752 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:57.494287 27752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:57.494443 27752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:57.495428 27752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "2a07f85745ee47a7b30264d1c1dbb4d3"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "5528ffbd897d9ff9636bc49ade6deb25"
server_key_iv: "6889e853c11c86ed9d4bbd21c535c487"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.495775 27752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "2a07f85745ee47a7b30264d1c1dbb4d3"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "5528ffbd897d9ff9636bc49ade6deb25"
server_key_iv: "6889e853c11c86ed9d4bbd21c535c487"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.497007 27752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:07:57.497720 27767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.497888 27752 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.497975 27752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "2a07f85745ee47a7b30264d1c1dbb4d3"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "5528ffbd897d9ff9636bc49ade6deb25"
server_key_iv: "6889e853c11c86ed9d4bbd21c535c487"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.498044 27752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:57.516086 27752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:57.517822 27752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:57.517967 27752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:57.521634 27752 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:45369
I20260925 05:07:57.521664 27819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:45369 every 8 connection(s)
I20260925 05:07:57.521974 27752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:07:57.522573 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27752
I20260925 05:07:57.522692 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:07:57.522899 27820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:57.522915 14468 external_mini_cluster.cc:1428] Setting key 7f02d597a357b5d34941eeb0f447c10f
I20260925 05:07:57.524966 27820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.33.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.61@KRBTEST.COM" created.
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:57.535463 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.525652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.14.33.62:45369)
0925 05:07:57.525835 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.525844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.525937 (+ 93us) server_negotiation.cc:407] Connection header received
0925 05:07:57.526494 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.526497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.526522 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.526619 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.527149 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.527921 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.528775 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.529075 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.529208 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.532556 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.532563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.532563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.532569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.534028 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.534388 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.534390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.534391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.534466 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.534662 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.534663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.534663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.534872 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.534958 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.535121 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.535248 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260925 05:07:57.535465 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.525639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45369 (local address 127.0.0.1:43248)
0925 05:07:57.525808 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.525845 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.526347 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.526635 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.526637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.526908 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.527043 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.527048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.528030 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.528031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.528676 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.528678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.529182 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.529183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.529243 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.529729 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.529739 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.532475 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.534122 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.534128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.534130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.534304 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.534527 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.534528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.534528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.534605 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.534941 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.534946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.535001 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.535098 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.535313 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":90,"thread_start_us":85,"threads_started":1}
I20260925 05:07:57.536059 27820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:07:57.536369 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.536262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45802)
0925 05:07:57.536304 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.536342 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 05:07:57.536482 27768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
I20260925 05:07:57.536856 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.536772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45804)
0925 05:07:57.536814 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.536836 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:07:57.536955 27820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:45101: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab.
Entry for principal kudu/127.14.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab.
I20260925 05:07:57.545302 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:45101
--webserver_interface=127.14.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab
--principal=kudu/127.14.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:45369,127.14.33.61:45101,127.14.33.60:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:57.586164 27820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } attempt: 1
I20260925 05:07:57.586666 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.586538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45810)
0925 05:07:57.586598 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.586638 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:07:57.587018 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.586943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45818)
0925 05:07:57.586979 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.587003 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 05:07:57.587116 27820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:45101: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
W20260925 05:07:57.622287 27827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:57.622491 27827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:57.622512 27827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:57.623536 27827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:57.623567 27827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:57.623581 27827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:57.623594 27827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:57.623606 27827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:57.624977 27827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:45369,127.14.33.61:45101,127.14.33.60:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab
--principal=kudu/127.14.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:57.625208 27827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:57.625396 27827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:07:57.629832 27835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.629854 27833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.629837 27832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.630141 27827 server_base.cc:1034] running on GCE node
I20260925 05:07:57.630414 27827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:57.630688 27827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:57.631855 27827 hybrid_clock.cc:648] HybridClock initialized: now 1790312877631838 us; error 32 us; skew 500 ppm
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:57.634266 27827 init.cc:377] Logged in from keytab as kudu/127.14.33.61@KRBTEST.COM (short username kudu)
I20260925 05:07:57.634652 27827 webserver.cc:460] Webserver started at http://127.14.33.61:42133/ using document root <none> and password file <none>
I20260925 05:07:57.634867 27827 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:57.634932 27827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:57.635073 27827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:57.636073 27827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data/instance:
uuid: "3584ef0599e24a9d84da2cc798dfa39e"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "fc58aefeacf0a111106c20eb9a42cc57"
server_key_iv: "65aefd7b39570dae8df62ff44e8c1b32"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.636430 27827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal/instance:
uuid: "3584ef0599e24a9d84da2cc798dfa39e"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "fc58aefeacf0a111106c20eb9a42cc57"
server_key_iv: "65aefd7b39570dae8df62ff44e8c1b32"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.637784 27827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 05:07:57.638439 27842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.638603 27827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:07:57.638705 27827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal
uuid: "3584ef0599e24a9d84da2cc798dfa39e"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "fc58aefeacf0a111106c20eb9a42cc57"
server_key_iv: "65aefd7b39570dae8df62ff44e8c1b32"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.638778 27827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:57.655329 27820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } attempt: 2
I20260925 05:07:57.655750 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.655608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45832)
0925 05:07:57.655687 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.655725 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 05:07:57.656068 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.655976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45834)
0925 05:07:57.656021 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.656044 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 05:07:57.656165 27820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:45101: Network error: Client connection negotiation failed: client connection to 127.14.33.61:45101: connect: Connection refused (error 111)
I20260925 05:07:57.665707 27827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:57.667375 27827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:57.667536 27827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:57.671221 27827 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.61:45101
I20260925 05:07:57.671252 27894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.61:45101 every 8 connection(s)
I20260925 05:07:57.671607 27827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
I20260925 05:07:57.672399 27895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:57.673848 27895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:57.679776 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27827
I20260925 05:07:57.679903 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-1/wal/instance
I20260925 05:07:57.680052 14468 external_mini_cluster.cc:1428] Setting key d67284d486da8b3b3a460ac1b068e67d
I20260925 05:07:57.682150 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.674476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.14.33.62:45369)
0925 05:07:57.674550 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.674552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.674696 (+ 144us) server_negotiation.cc:407] Connection header received
0925 05:07:57.675095 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.675096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.675105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.675149 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.675667 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.676111 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.676972 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.677161 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.677207 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.679029 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.679030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.679030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.679035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.680462 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.680913 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.680914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.680914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.680972 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.681299 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.681300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.681300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.681605 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.681718 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.681986 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.682065 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 05:07:57.682329 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.674430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45369 (local address 127.0.0.1:43262)
0925 05:07:57.674593 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.674624 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.675022 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.675205 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.675208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.675430 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.675603 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.675607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.676183 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.676183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.676907 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.676909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.677223 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.677224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.677324 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.677755 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.677763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.678954 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.680563 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.680570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.680572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.680816 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.681043 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.681044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.681044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.681221 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.681708 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.681715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.681851 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.681963 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.682105 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260925 05:07:57.682817 27895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.60@KRBTEST.COM" created.
I20260925 05:07:57.690563 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.683037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45836)
0925 05:07:57.683275 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.683285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.683325 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.683493 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.683494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.683645 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.683701 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.683703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.684500 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.684501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.685384 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.685387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.685938 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.685939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.686012 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.686331 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.686336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.687537 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.689385 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.689386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.689386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.689612 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.689879 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.689880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.689880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.689938 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.690266 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.690266 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.690302 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.690373 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.690462 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260925 05:07:57.690665 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.683097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.14.33.61:45101)
0925 05:07:57.683338 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.683342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.683364 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:07:57.683385 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.683387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.683399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.683472 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.683778 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.684409 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.685469 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.685809 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.685938 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.687637 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.687646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.687646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.687653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.689293 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.689723 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.689725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.689726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.689778 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.690045 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.690046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.690046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.690192 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.690287 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.690403 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.690529 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":185,"thread_start_us":130,"threads_started":1}
I20260925 05:07:57.691164 27895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:07:57.691599 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.691487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35654)
0925 05:07:57.691532 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.691570 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:07:57.691761 27845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
I20260925 05:07:57.692296 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.692103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35670)
0925 05:07:57.692234 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.692269 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 05:07:57.692438 27895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:45213: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab.
Entry for principal kudu/127.14.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab.
I20260925 05:07:57.704479 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:45213
--webserver_interface=127.14.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab
--principal=kudu/127.14.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:45369,127.14.33.61:45101,127.14.33.60:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:07:57.741628 27895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } attempt: 1
I20260925 05:07:57.742093 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.741931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35674)
0925 05:07:57.742021 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.742058 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 05:07:57.742554 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.742447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35680)
0925 05:07:57.742500 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.742527 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:07:57.742669 27895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:45213: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
I20260925 05:07:57.747313 27820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } attempt: 3
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:07:57.753790 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.747597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45838)
0925 05:07:57.747635 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.747645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.747714 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.747852 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.747853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.747996 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.748072 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.748074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.748717 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.748718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.749272 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.749274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.749606 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.749607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.749687 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.750097 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.750104 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.751237 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.752746 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.752747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.752747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.752941 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.753171 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.753172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.753172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.753246 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.753481 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.753482 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.753523 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.753609 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.753699 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 05:07:57.753786 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.747640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.14.33.61:45101)
0925 05:07:57.747712 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.747714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.747724 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:07:57.747769 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.747769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.747778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.747847 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.748149 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.748626 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.749348 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.749540 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.749610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.751319 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.751321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.751321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.751328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.752647 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.753025 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.753027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.753027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.753081 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.753309 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.753310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.753310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.753414 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.753488 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.753633 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.753705 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 05:07:57.754184 27820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:07:57.754494 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.754393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35684)
0925 05:07:57.754448 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.754472 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:07:57.754776 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.754700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35686)
0925 05:07:57.754738 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.754758 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 05:07:57.754858 27820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:45213: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
W20260925 05:07:57.783177 27902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:07:57.783372 27902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:07:57.783392 27902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:07:57.784519 27902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:07:57.784559 27902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:07:57.784572 27902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:07:57.784621 27902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:07:57.784638 27902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:07:57.785988 27820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } attempt: 1
I20260925 05:07:57.786206 27902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:45369,127.14.33.61:45101,127.14.33.60:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab
--principal=kudu/127.14.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:07:57.786357 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.786237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35696)
0925 05:07:57.786300 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.786334 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:07:57.786454 27902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:07:57.786648 27902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:07:57.786662 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.786580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35698)
0925 05:07:57.786608 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.786630 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 05:07:57.786747 27820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:45213: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
W20260925 05:07:57.791215 27908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.791347 27902 server_base.cc:1034] running on GCE node
W20260925 05:07:57.791234 27907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:07:57.791230 27910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:07:57.791723 27902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:07:57.792002 27902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:07:57.793179 27902 hybrid_clock.cc:648] HybridClock initialized: now 1790312877793160 us; error 44 us; skew 500 ppm
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:07:57.795794 27902 init.cc:377] Logged in from keytab as kudu/127.14.33.60@KRBTEST.COM (short username kudu)
I20260925 05:07:57.796200 27902 webserver.cc:460] Webserver started at http://127.14.33.60:44309/ using document root <none> and password file <none>
I20260925 05:07:57.796419 27902 fs_manager.cc:359] Metadata directory not provided
I20260925 05:07:57.796482 27902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:07:57.796629 27902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:07:57.797701 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data/instance:
uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "b54a8a8c9f575e60a32be09e1c06752b"
server_key_iv: "48c10ee050899d2e46ead25d78b949bb"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.798100 27902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal/instance:
uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "b54a8a8c9f575e60a32be09e1c06752b"
server_key_iv: "48c10ee050899d2e46ead25d78b949bb"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.799351 27902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:07:57.800107 27917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:07:57.800307 27902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:07:57.800395 27902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal
uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24"
format_stamp: "Formatted at 2026-09-25 05:07:57 on dist-test-slave-4dmq"
server_key: "b54a8a8c9f575e60a32be09e1c06752b"
server_key_iv: "48c10ee050899d2e46ead25d78b949bb"
server_key_version: "encryptionkey@0"
I20260925 05:07:57.800462 27902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:07:57.810877 27895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } attempt: 2
I20260925 05:07:57.811242 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.811132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35702)
0925 05:07:57.811185 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.811218 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:07:57.811533 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.811448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35714)
0925 05:07:57.811490 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.811511 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 05:07:57.811604 27895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:45213: Network error: Client connection negotiation failed: client connection to 127.14.33.60:45213: connect: Connection refused (error 111)
I20260925 05:07:57.819831 27902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:07:57.821584 27902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:07:57.821738 27902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:07:57.825275 27902 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.60:45213
I20260925 05:07:57.825307 27969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.60:45213 every 8 connection(s)
I20260925 05:07:57.825609 27902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
I20260925 05:07:57.826421 27970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:07:57.827795 27970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:07:57.828588 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 27902
I20260925 05:07:57.828711 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312729583056-14468-0/minicluster-data/master-2/wal/instance
I20260925 05:07:57.828876 14468 external_mini_cluster.cc:1428] Setting key 9f60a0a6b57d744a8901cab4362c5f01
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:07:57.837083 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.828372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.14.33.62:45369)
0925 05:07:57.828419 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.828421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.828624 (+ 203us) server_negotiation.cc:407] Connection header received
0925 05:07:57.829288 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.829289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.829301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.829363 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.829905 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.830391 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.831247 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.831466 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.831549 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.833935 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.833937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.833937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.833945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.835683 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.836075 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.836076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.836077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.836138 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.836381 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.836382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.836383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.836506 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.836556 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.836912 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.836994 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:07:57.837208 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.828353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45369 (local address 127.0.0.1:43272)
0925 05:07:57.828514 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.828546 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.829181 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.829440 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.829443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.829658 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.829819 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.829822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.830470 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.830471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.831159 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.831161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.831558 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.831559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.831692 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.832164 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.832173 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.833849 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.835781 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.835787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.835788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.835996 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.836208 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.836209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.836209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.836305 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.836573 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.836579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.836728 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.836869 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.837010 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260925 05:07:57.837663 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.829225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.14.33.60:45213)
0925 05:07:57.829386 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.829389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.829524 (+ 135us) server_negotiation.cc:407] Connection header received
0925 05:07:57.829579 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.829581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.829609 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.829673 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.830267 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.830990 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.831716 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.831991 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.832066 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.834630 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.834637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.834637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.834644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.836310 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.836671 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.836673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.836673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.836722 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.836932 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.836933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.836933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.837099 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.837301 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.837492 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.837561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260925 05:07:57.837941 27970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:07:57.844553 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.838155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:45101 (local address 127.0.0.1:45850)
0925 05:07:57.838239 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.838248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.838284 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.838407 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.838407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.838556 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.838611 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.838612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.839196 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.839197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.839627 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.839628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.839978 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.839979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.840045 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.840382 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.840386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.841818 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.843595 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.843596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.843596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.843765 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.843972 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.843972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.843973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.844027 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.844280 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.844281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.844320 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.844387 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.844469 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 05:07:57.844560 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.838243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.14.33.61:45101)
0925 05:07:57.838301 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.838303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.838328 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:07:57.838357 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.838358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.838366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.838404 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.838679 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.839137 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.839712 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.839914 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.839962 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.841924 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.841925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.841925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.841932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.843507 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.843835 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.843837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.843837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.843900 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.844097 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.844098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.844099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.844214 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.844288 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.844414 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.844481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:07:57.845177 27970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:07:57.846498 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.838573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.14.33.62:45369)
0925 05:07:57.838612 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.838614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.838622 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:07:57.839207 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.839208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.839216 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.839280 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.839741 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.840263 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.840978 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.841287 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.841349 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.843569 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.843570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.843571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.843577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.845197 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.845692 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.845693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.845693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.845744 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.845982 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.845983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.845983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.846115 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.846308 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.846346 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.846411 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:07:57.852133 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.845424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35726)
0925 05:07:57.845471 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.845481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.845524 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.845640 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.845640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.845762 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.845783 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.845784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.846389 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.846390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.846961 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.846963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.847357 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.847358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.847420 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.847719 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.847724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.848854 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.850729 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.850730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.850730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.850880 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.851562 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.851563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.851563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.851612 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.851873 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.851874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.851914 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.851995 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.852082 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 05:07:57.852233 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.845472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.14.33.60:45213)
0925 05:07:57.845504 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.845506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.845512 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:07:57.845577 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.845578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.845585 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.845636 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.845855 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.846315 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.847053 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.847273 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.847355 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.848944 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.848945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.848945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.848951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.850636 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.851420 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.851421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.851421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.851476 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.851672 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.851673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.851674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.851808 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.851867 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.852030 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.852143 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":20}
I20260925 05:07:57.853713 27970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: Bootstrap starting.
I20260925 05:07:57.854544 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.847348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.14.33.61:45101)
0925 05:07:57.847388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.847390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.847396 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:07:57.847413 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.847414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.847422 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.847470 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.848044 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.848473 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.849230 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.849403 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.849453 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.851817 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.851818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.851818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.851824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.853551 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.853917 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.853918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.853918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.853975 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.854173 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.854173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.854173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.854252 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.854367 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.854447 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.854495 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:07:57.854717 27970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:57.854962 27970 log.cc:824] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:57.855744 27970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: No bootstrap required, opened a new log
I20260925 05:07:57.856165 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:07:57.857417 27970 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.857569 27970 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:57.857609 27970 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd9f1a5823b46008daf3fbf4fa6aa24, State: Initialized, Role: FOLLOWER
I20260925 05:07:57.857739 27970 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.858048 27970 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:07:57.859704 27994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:07:57.859758 27994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:07:57.861711 27983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.861871 27820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } attempt: 2
I20260925 05:07:57.861995 27983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:57.859887 27902 master_runner.cc:186] Error getting master registration for 127.14.33.62:45369: OK, instance_id { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" instance_seqno: 1790312877520591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:07:57.864041 27902 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:07:57.864745 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.857746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.14.33.62:45369)
0925 05:07:57.857791 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.857793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.857896 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:07:57.857971 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.857972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.857983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.858010 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.858594 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.859125 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.860044 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.860289 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.860342 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.861813 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.861814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.861814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.861820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.863343 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.863755 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.863756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.863757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.863810 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.864013 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.864014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.864015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.864140 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.864226 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.864554 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.864628 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:07:57.866969 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.857665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.14.33.61:45101)
0925 05:07:57.857700 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.857702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.859451 (+ 1749us) server_negotiation.cc:407] Connection header received
0925 05:07:57.859902 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.859903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.859912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.859939 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.860554 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.861177 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.862280 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.862547 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.862601 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.863858 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.863859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.863860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.863866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.865458 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.865942 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.865943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.865943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.865990 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.866178 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.866179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.866179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.866311 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.866404 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.866752 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.866877 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:07:57.868849 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.857706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.14.33.60:45213)
0925 05:07:57.857787 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.857789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.857797 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:07:57.857849 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.857850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.857861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.857921 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.858550 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.859224 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.860444 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.860685 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.860719 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.864866 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.864867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.864867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.864874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.866645 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.867049 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.867050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.867050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.867120 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.868175 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.868176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.868177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.868354 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.868485 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.868615 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.868731 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 05:07:57.870048 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.862052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35740)
0925 05:07:57.862134 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.862142 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.862179 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.862401 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.862402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.862555 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.862591 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.862592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.863587 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.863588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.864362 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.864364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.865209 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.865210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.865273 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.865681 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.865686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.867162 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.868864 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.868865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.868866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.869048 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.869414 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.869415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.869416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.869472 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.869736 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.869737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.869786 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.869873 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.869958 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 05:07:57.870085 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.862136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.14.33.60:45213)
0925 05:07:57.862301 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.862303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.862308 (+ 5us) server_negotiation.cc:407] Connection header received
0925 05:07:57.862321 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.862321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.862328 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.862650 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.862690 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.863468 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.864458 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.864795 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.865406 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.867279 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.867280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.867280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.867287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.868776 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.869277 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.869277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.869278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.869347 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.869547 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.869548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.869548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.869668 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.869753 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.869898 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.869967 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
I20260925 05:07:57.871423 27820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3: Bootstrap starting.
I20260925 05:07:57.871989 27820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:57.872267 27820 log.cc:824] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:57.873250 27820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3: No bootstrap required, opened a new log
I20260925 05:07:57.875209 27820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.875336 27820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:57.875367 27820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a07f85745ee47a7b30264d1c1dbb4d3, State: Initialized, Role: FOLLOWER
I20260925 05:07:57.875471 27820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.875759 27998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.875815 27820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:07:57.875890 27998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:07:57.877671 28009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:07:57.877722 28009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:07:57.878427 27752 master_runner.cc:186] Error getting master registration for 127.14.33.61:45101: OK, instance_id { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" instance_seqno: 1790312877670218 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:07:57.880326 27752 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:07:57.902762 27895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } attempt: 3
Sep 25 05:07:57 dist-test-slave-4dmq krb5kdc[27739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312877, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:07:57.910311 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.903005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:45213 (local address 127.0.0.1:35742)
0925 05:07:57.903047 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:07:57.903057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:07:57.903088 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:07:57.903264 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:07:57.903265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:07:57.903446 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:07:57.903495 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.903497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.904405 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.904406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.905155 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:07:57.905157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.905630 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.905631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:07:57.905688 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.906132 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:07:57.906137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:07:57.907466 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:07:57.909180 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.909181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.909181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.909418 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.909623 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:07:57.909624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:07:57.909624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:07:57.909690 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:07:57.909989 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:07:57.909990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:07:57.910037 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:07:57.910110 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:07:57.910201 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 05:07:57.910329 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:07:57.903067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.14.33.60:45213)
0925 05:07:57.903156 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 05:07:57.903158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:07:57.903164 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:07:57.903185 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:07:57.903185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:07:57.903195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:07:57.903241 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:07:57.903574 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.904329 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.905251 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:07:57.905557 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:07:57.905623 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:07:57.907551 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:07:57.907552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:07:57.907552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:07:57.907559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:07:57.909083 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.909494 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.909495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.909496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.909553 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:07:57.909756 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:07:57.909757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:07:57.909757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:07:57.909901 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:07:57.909968 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:07:57.910147 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 05:07:57.910239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 05:07:57.912181 27895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: Bootstrap starting.
I20260925 05:07:57.912889 27895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: Neither blocks nor log segments found. Creating new log.
I20260925 05:07:57.913249 27895 log.cc:824] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:07:57.914064 27895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: No bootstrap required, opened a new log
I20260925 05:07:57.915951 27895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.916079 27895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:07:57.916112 27895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3584ef0599e24a9d84da2cc798dfa39e, State: Initialized, Role: FOLLOWER
I20260925 05:07:57.916252 27895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.916500 28012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.916564 27895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:07:57.918385 28023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:07:57.918438 28023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 05:07:57.918067 27848 tablet.cc:2369] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:07:57.921401 27827 master_runner.cc:263] No leader master found from master 3584ef0599e24a9d84da2cc798dfa39e
I20260925 05:07:57.921443 27827 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:07:57.921550 28012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:57.930356 27998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:07:57.930436 27998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.930655 27998 leader_election.cc:288] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3584ef0599e24a9d84da2cc798dfa39e (127.14.33.61:45101), fbd9f1a5823b46008daf3fbf4fa6aa24 (127.14.33.60:45213)
I20260925 05:07:57.931370 27869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3584ef0599e24a9d84da2cc798dfa39e" is_pre_election: true
I20260925 05:07:57.931489 27869 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a07f85745ee47a7b30264d1c1dbb4d3 in term 0.
I20260925 05:07:57.931612 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" is_pre_election: true
I20260925 05:07:57.931736 27944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a07f85745ee47a7b30264d1c1dbb4d3 in term 0.
I20260925 05:07:57.932207 27770 leader_election.cc:302] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a07f85745ee47a7b30264d1c1dbb4d3, fbd9f1a5823b46008daf3fbf4fa6aa24; no voters:
I20260925 05:07:57.932608 27998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:07:57.932662 27998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:07:57.932684 27998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:57.933616 27998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.933718 27998 leader_election.cc:288] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [CANDIDATE]: Term 1 election: Requested vote from peers 3584ef0599e24a9d84da2cc798dfa39e (127.14.33.61:45101), fbd9f1a5823b46008daf3fbf4fa6aa24 (127.14.33.60:45213)
I20260925 05:07:57.934033 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24"
I20260925 05:07:57.934099 27944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:57.934940 27944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a07f85745ee47a7b30264d1c1dbb4d3 in term 1.
I20260925 05:07:57.935128 27770 leader_election.cc:302] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a07f85745ee47a7b30264d1c1dbb4d3, fbd9f1a5823b46008daf3fbf4fa6aa24; no voters:
I20260925 05:07:57.935293 27869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3584ef0599e24a9d84da2cc798dfa39e"
I20260925 05:07:57.935353 27869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:07:57.936102 27869 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a07f85745ee47a7b30264d1c1dbb4d3 in term 1.
I20260925 05:07:57.936340 27998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:07:57.936491 27998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2a07f85745ee47a7b30264d1c1dbb4d3, State: Running, Role: LEADER
I20260925 05:07:57.936594 27998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } }
I20260925 05:07:57.936942 27998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a07f85745ee47a7b30264d1c1dbb4d3. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.937001 27998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:57.937296 28027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:07:57.937494 28027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:07:57.939301 27944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [term 1 FOLLOWER]: Refusing update from remote peer 2a07f85745ee47a7b30264d1c1dbb4d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:57.939625 27998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:57.940825 27983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a07f85745ee47a7b30264d1c1dbb4d3. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.940903 27983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:57.941362 27869 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [term 1 FOLLOWER]: Refusing update from remote peer 2a07f85745ee47a7b30264d1c1dbb4d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:07:57.941709 27998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:07:57.942925 28012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a07f85745ee47a7b30264d1c1dbb4d3. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.943011 28012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:57.945281 27998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.945370 27998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:57.945926 28012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.946005 28012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:57.946336 28028 mvcc.cc:205] Tried to move back new op lower bound from 7333121548037230592 to 7333121548028768256. Current Snapshot: MvccSnapshot[applied={T|T < 7333121548037230592}]
I20260925 05:07:57.946470 28031 mvcc.cc:205] Tried to move back new op lower bound from 7333121548037230592 to 7333121548028768256. Current Snapshot: MvccSnapshot[applied={T|T < 7333121548037230592}]
I20260925 05:07:57.947386 27998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.947461 27998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3 [sys.catalog]: This master's current role is: LEADER
I20260925 05:07:57.947840 27983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a07f85745ee47a7b30264d1c1dbb4d3" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45369 } } peers { permanent_uuid: "3584ef0599e24a9d84da2cc798dfa39e" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 45101 } } peers { permanent_uuid: "fbd9f1a5823b46008daf3fbf4fa6aa24" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 45213 } } }
I20260925 05:07:57.947921 27983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:07:57.948582 28029 mvcc.cc:205] Tried to move back new op lower bound from 7333121548037230592 to 7333121548028768256. Current Snapshot: MvccSnapshot[applied={T|T < 7333121548037230592}]
I20260925 05:07:57.949460 28027 catalog_manager.cc:1347] Generated new cluster ID: b8d470a64d954eefa89a08537543c014
I20260925 05:07:57.949506 28027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:07:57.970286 28027 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:07:57.971091 28027 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:07:57.990382 28027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a07f85745ee47a7b30264d1c1dbb4d3: Generated new TSK 0
I20260925 05:07:57.990703 28027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:07:58.860155 27994 catalog_manager.cc:1259] Loaded cluster ID: b8d470a64d954eefa89a08537543c014
I20260925 05:07:58.860242 27994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: loading cluster ID for follower catalog manager: success
I20260925 05:07:58.860916 27994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: acquiring CA information for follower catalog manager: success
I20260925 05:07:58.861202 27994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbd9f1a5823b46008daf3fbf4fa6aa24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:07:58.918839 28023 catalog_manager.cc:1259] Loaded cluster ID: b8d470a64d954eefa89a08537543c014
I20260925 05:07:58.918911 28023 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: loading cluster ID for follower catalog manager: success
I20260925 05:07:58.919623 28023 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: acquiring CA information for follower catalog manager: success
I20260925 05:07:58.919858 28023 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3584ef0599e24a9d84da2cc798dfa39e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:07:59.045675 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27752
I20260925 05:07:59.050899 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27827
I20260925 05:07:59.055405 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 27902
2026-09-25T05:07:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2760 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:07:59 dist-test-slave-4dmq krb5kdc[28043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:07:59 dist-test-slave-4dmq krb5kdc[28043](info): set up 2 sockets
Sep 25 05:07:59 dist-test-slave-4dmq krb5kdc[28043](info): commencing operation
krb5kdc: starting...
W20260925 05:08:00.070356 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:08:00 dist-test-slave-4dmq krb5kdc[28043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:08:00.177418 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41051
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:00.255779 28056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:00.255991 28056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:00.256026 28056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:00.257205 28056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:00.257263 28056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:00.257284 28056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:00.257310 28056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:00.257335 28056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:00.258906 28056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:00.259199 28056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:00.259435 28056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:00.264132 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:00.264201 28064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:00.264132 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:00.264359 28056 server_base.cc:1034] running on GCE node
I20260925 05:08:00.264712 28056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:00.264981 28056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:00.266124 28056 hybrid_clock.cc:648] HybridClock initialized: now 1790312880266101 us; error 59 us; skew 500 ppm
Sep 25 05:08:00 dist-test-slave-4dmq krb5kdc[28043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:00.268580 28056 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:08:00.268989 28056 webserver.cc:460] Webserver started at http://127.14.33.62:46579/ using document root <none> and password file <none>
I20260925 05:08:00.269212 28056 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:00.269254 28056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:00.269338 28056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:00.270252 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "b51a183653e34771a3b95a00569fb566"
format_stamp: "Formatted at 2026-09-25 05:08:00 on dist-test-slave-4dmq"
server_key: "4c45000681ad4ea4abd3ff5d6c888766"
server_key_iv: "2eab9b7b107e1cde7ffdbb02fedb82c2"
server_key_version: "encryptionkey@0"
I20260925 05:08:00.270533 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "b51a183653e34771a3b95a00569fb566"
format_stamp: "Formatted at 2026-09-25 05:08:00 on dist-test-slave-4dmq"
server_key: "4c45000681ad4ea4abd3ff5d6c888766"
server_key_iv: "2eab9b7b107e1cde7ffdbb02fedb82c2"
server_key_version: "encryptionkey@0"
I20260925 05:08:00.271688 28056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:00.272342 28071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:00.272507 28056 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:00.272604 28056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "b51a183653e34771a3b95a00569fb566"
format_stamp: "Formatted at 2026-09-25 05:08:00 on dist-test-slave-4dmq"
server_key: "4c45000681ad4ea4abd3ff5d6c888766"
server_key_iv: "2eab9b7b107e1cde7ffdbb02fedb82c2"
server_key_version: "encryptionkey@0"
I20260925 05:08:00.272686 28056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:00.291275 28056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:00.292960 28056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:00.293180 28056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:00.297088 28056 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:41051
I20260925 05:08:00.297163 28123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:41051 every 8 connection(s)
I20260925 05:08:00.297525 28056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:00.298324 28124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:00.300465 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566: Bootstrap starting.
I20260925 05:08:00.301045 28124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:00.301380 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28056
I20260925 05:08:00.301391 28124 log.cc:824] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:00.301462 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:08:00.301687 14468 external_mini_cluster.cc:1428] Setting key 666f2a2cab87648e81f9d57746a2ad4c
I20260925 05:08:00.302280 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566: No bootstrap required, opened a new log
I20260925 05:08:00.304448 28124 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51a183653e34771a3b95a00569fb566" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41051 } }
I20260925 05:08:00.304616 28124 raft_consensus.cc:374] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:00.304664 28124 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51a183653e34771a3b95a00569fb566, State: Initialized, Role: FOLLOWER
I20260925 05:08:00.304795 28124 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51a183653e34771a3b95a00569fb566" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41051 } }
I20260925 05:08:00.304898 28124 raft_consensus.cc:388] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:00.304949 28124 raft_consensus.cc:482] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:00.304998 28124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:00.305572 28124 raft_consensus.cc:504] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51a183653e34771a3b95a00569fb566" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41051 } }
I20260925 05:08:00.305697 28124 leader_election.cc:302] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b51a183653e34771a3b95a00569fb566; no voters:
I20260925 05:08:00.305874 28124 leader_election.cc:288] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:08:00.305913 28129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:00.306104 28129 raft_consensus.cc:686] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [term 1 LEADER]: Becoming Leader. State: Replica: b51a183653e34771a3b95a00569fb566, State: Running, Role: LEADER
I20260925 05:08:00.306111 28124 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:00.306221 28129 consensus_queue.cc:237] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51a183653e34771a3b95a00569fb566" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41051 } }
I20260925 05:08:00.306614 28129 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b51a183653e34771a3b95a00569fb566. Latest consensus state: current_term: 1 leader_uuid: "b51a183653e34771a3b95a00569fb566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51a183653e34771a3b95a00569fb566" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41051 } } }
I20260925 05:08:00.306739 28129 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:00.306777 28130 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b51a183653e34771a3b95a00569fb566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51a183653e34771a3b95a00569fb566" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41051 } } }
I20260925 05:08:00.306931 28130 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566 [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:00.306982 28138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:00.307492 28138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:08:00 dist-test-slave-4dmq krb5kdc[28043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:00.309093 28138 catalog_manager.cc:1347] Generated new cluster ID: 81f34ed6207848d8bc02944cf13e9ac5
I20260925 05:08:00.309195 28138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:08:00.312318 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:00.302182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.14.33.62:41051)
0925 05:08:00.302358 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:00.302367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:00.302399 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:08:00.302806 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:00.302812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:00.302856 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:00.303017 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:00.303639 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.304512 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.305482 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:00.306030 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:00.306450 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:00.309045 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:00.309049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:00.309049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:00.309060 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:00.310826 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:00.311188 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:00.311191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:00.311191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:00.311288 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:00.311501 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:00.311502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:00.311503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:00.311730 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:00.311912 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:00.311988 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:08:00.312116 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260925 05:08:00.313232 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:08:00.325459 28138 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:08:00.325947 28138 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:00.333006 28138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b51a183653e34771a3b95a00569fb566: Generated new TSK 0
I20260925 05:08:00.333274 28138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:08:00.366923 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28056
2026-09-25T05:08:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1310 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:08:00 dist-test-slave-4dmq krb5kdc[28151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:08:00 dist-test-slave-4dmq krb5kdc[28151](info): set up 2 sockets
Sep 25 05:08:00 dist-test-slave-4dmq krb5kdc[28151](info): commencing operation
krb5kdc: starting...
W20260925 05:08:01.354338 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:08:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:08:01.459899 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40555
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:01.535492 28164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:01.535670 28164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:01.535691 28164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:01.536665 28164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:01.536697 28164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:01.536710 28164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:01.536720 28164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:01.536731 28164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:01.538211 28164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:01.538511 28164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:01.538714 28164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:01.543403 28172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:01.543440 28170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:01.543579 28169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:01.543577 28164 server_base.cc:1034] running on GCE node
I20260925 05:08:01.543843 28164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:01.544083 28164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:01.545184 28164 hybrid_clock.cc:648] HybridClock initialized: now 1790312881545166 us; error 27 us; skew 500 ppm
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:01.547621 28164 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:08:01.548123 28164 webserver.cc:460] Webserver started at http://127.14.33.62:38269/ using document root <none> and password file <none>
I20260925 05:08:01.548357 28164 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:01.548430 28164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:01.548578 28164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:01.549654 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "00d222d8b08445f7a31a13b8de0edb4a"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "bc78719d8c070b20cb112894fc8f2856"
server_key_iv: "218743524cb0c85fed817b2dd5fb4f2a"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.549989 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "00d222d8b08445f7a31a13b8de0edb4a"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "bc78719d8c070b20cb112894fc8f2856"
server_key_iv: "218743524cb0c85fed817b2dd5fb4f2a"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.551288 28164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:01.552085 28179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.552280 28164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:01.552385 28164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "00d222d8b08445f7a31a13b8de0edb4a"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "bc78719d8c070b20cb112894fc8f2856"
server_key_iv: "218743524cb0c85fed817b2dd5fb4f2a"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.552457 28164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:01.560020 28164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:01.562163 28164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:01.562322 28164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:01.566363 28164 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40555
I20260925 05:08:01.566403 28231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40555 every 8 connection(s)
I20260925 05:08:01.566721 28164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:01.567600 28232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:01.569672 28232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap starting.
I20260925 05:08:01.570252 28232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:01.570565 28232 log.cc:824] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:01.571234 28232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: No bootstrap required, opened a new log
I20260925 05:08:01.572731 28232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:01.572854 28232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:01.572875 28232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d222d8b08445f7a31a13b8de0edb4a, State: Initialized, Role: FOLLOWER
I20260925 05:08:01.573042 28232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:01.573100 28232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:01.573180 28232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:01.573235 28232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:01.573886 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28164
I20260925 05:08:01.574033 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:08:01.574231 14468 external_mini_cluster.cc:1428] Setting key 96525bb7a62d210ae13b02bed6a5027c
I20260925 05:08:01.574298 28232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:01.574398 28232 leader_election.cc:302] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d222d8b08445f7a31a13b8de0edb4a; no voters:
I20260925 05:08:01.574553 28232 leader_election.cc:288] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:08:01.574570 28235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:01.574723 28235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 LEADER]: Becoming Leader. State: Replica: 00d222d8b08445f7a31a13b8de0edb4a, State: Running, Role: LEADER
I20260925 05:08:01.574834 28232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:01.574851 28235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:01.575227 28237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d222d8b08445f7a31a13b8de0edb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } } }
I20260925 05:08:01.575309 28237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:01.575505 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d222d8b08445f7a31a13b8de0edb4a. Latest consensus state: current_term: 1 leader_uuid: "00d222d8b08445f7a31a13b8de0edb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } } }
I20260925 05:08:01.575619 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: This master's current role is: LEADER
W20260925 05:08:01.576498 28253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:08:01.576558 28253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:08:01.576601 28243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:01.576727 28243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:01.578167 28243 catalog_manager.cc:1347] Generated new cluster ID: e674cb03d23a4d67b1e8b2cb4dd93375
I20260925 05:08:01.578217 28243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:01.585994 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.574769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.14.33.62:40555)
0925 05:08:01.575134 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:01.575142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:01.575189 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:08:01.575314 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:01.575321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:01.575375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:01.576326 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:01.576423 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.577661 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.578668 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.579212 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.579315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.581961 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:01.581965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:01.581965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:01.581977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:01.584411 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:01.584790 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:01.584793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:01.584793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:01.584918 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:01.585109 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:01.585110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:01.585110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:01.585438 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:01.585598 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:01.585660 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 05:08:01.585796 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":64,"threads_started":1}
I20260925 05:08:01.588109 28243 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:08:01.588541 28243 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:01.593166 28243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Generated new TSK 0
I20260925 05:08:01.593315 28243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:08:01.616503 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40555
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:01.691998 28258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:01.692201 28258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:01.692233 28258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:01.693353 28258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:01.693418 28258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:01.693462 28258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:08:01.694922 28258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:01.695170 28258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:01.695401 28258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:01.700119 28263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:01.700140 28264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:01.700119 28266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:01.700148 28258 server_base.cc:1034] running on GCE node
I20260925 05:08:01.700500 28258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:01.700716 28258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:01.701867 28258 hybrid_clock.cc:648] HybridClock initialized: now 1790312881701864 us; error 37 us; skew 500 ppm
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:01.704366 28258 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:08:01.704811 28258 webserver.cc:460] Webserver started at http://127.14.33.1:33441/ using document root <none> and password file <none>
I20260925 05:08:01.705039 28258 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:01.705101 28258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:01.705265 28258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:01.706198 28258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "f2456116ca0b427784d3057b261ca566"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "ab4747f3e9df893cab018a0ed4b48c52"
server_key_iv: "1b1be04efde391a85e2f7b827edc7bba"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.706566 28258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "f2456116ca0b427784d3057b261ca566"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "ab4747f3e9df893cab018a0ed4b48c52"
server_key_iv: "1b1be04efde391a85e2f7b827edc7bba"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.707890 28258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:01.708577 28273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.708787 28258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:01.708874 28258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "f2456116ca0b427784d3057b261ca566"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "ab4747f3e9df893cab018a0ed4b48c52"
server_key_iv: "1b1be04efde391a85e2f7b827edc7bba"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.708952 28258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:01.730077 28258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:01.731846 28258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:01.732003 28258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:01.732194 28258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:01.732491 28258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:01.732542 28258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.732580 28258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:01.732604 28258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.738186 28258 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:32803
I20260925 05:08:01.738233 28386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:32803 every 8 connection(s)
I20260925 05:08:01.738505 28258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:01.740882 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28258
I20260925 05:08:01.740985 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:08:01.741195 14468 external_mini_cluster.cc:1428] Setting key 816d6dd9c3f5a316812ba024fe9ea678
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:01.747641 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.739101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:55751 (local address 127.14.33.62:40555)
0925 05:08:01.739178 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:01.739181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:01.739332 (+ 151us) server_negotiation.cc:407] Connection header received
0925 05:08:01.739740 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:01.739741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:01.739769 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:01.739840 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:01.740667 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.741451 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.742476 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.742792 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.742865 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.745018 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:01.745019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:01.745020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:01.745030 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:01.746408 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:01.746790 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:01.746792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:01.746792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:01.746845 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:01.747075 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:01.747076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:01.747076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:01.747175 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:01.747245 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:01.747484 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 05:08:01.747558 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 05:08:01.747848 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.739068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:55751)
0925 05:08:01.739216 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:01.739247 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:01.739662 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:01.739886 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:01.739889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:01.740411 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:01.740591 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.740596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.741526 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.741527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.742390 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.742392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.742875 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.742876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.742965 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.743445 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:01.743455 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:01.744925 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:01.746499 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:01.746506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:01.746508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:01.746708 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:01.746913 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:01.746913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:01.746914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:01.747009 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:01.747238 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:01.747244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:01.747368 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 05:08:01.747464 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:08:01.747601 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260925 05:08:01.748291 28387 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40555
I20260925 05:08:01.748399 28387 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:01.748550 28387 heartbeater.cc:511] Master 127.14.33.62:40555 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
I20260925 05:08:01.749069 28196 ts_manager.cc:194] Registered new tserver with Master: f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803)
I20260925 05:08:01.749747 28196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:55751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:08:01.764531 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40555
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:01.841936 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:01.842104 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:01.842128 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:01.843127 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:01.843171 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:01.843218 28392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:08:01.844503 28392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:01.844734 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:01.844951 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:01.849390 28400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:01.849411 28398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:01.849537 28397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:01.849411 28392 server_base.cc:1034] running on GCE node
I20260925 05:08:01.849792 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:01.850054 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:01.851234 28392 hybrid_clock.cc:648] HybridClock initialized: now 1790312881851211 us; error 35 us; skew 500 ppm
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:01.853664 28392 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:08:01.854121 28392 webserver.cc:460] Webserver started at http://127.14.33.2:39589/ using document root <none> and password file <none>
I20260925 05:08:01.854342 28392 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:01.854405 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:01.854534 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:01.855567 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "302cf4b4efbb4b56af459cf52b189bc2"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "44ad4960467887dee6bc17b101711c4a"
server_key_iv: "85b07ec6a8cb394d46448bda1bb44fc9"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.855988 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "302cf4b4efbb4b56af459cf52b189bc2"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "44ad4960467887dee6bc17b101711c4a"
server_key_iv: "85b07ec6a8cb394d46448bda1bb44fc9"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.857359 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:08:01.858067 28407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.858244 28392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:01.858335 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "302cf4b4efbb4b56af459cf52b189bc2"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "44ad4960467887dee6bc17b101711c4a"
server_key_iv: "85b07ec6a8cb394d46448bda1bb44fc9"
server_key_version: "encryptionkey@0"
I20260925 05:08:01.858410 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:01.880424 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:01.882182 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:01.882334 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:01.882548 28392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:01.882866 28392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:01.882917 28392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.882949 28392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:01.882975 28392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:01.888532 28392 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:34855
I20260925 05:08:01.888581 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:34855 every 8 connection(s)
I20260925 05:08:01.888862 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:08:01 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:01.897609 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.889550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:51321 (local address 127.14.33.62:40555)
0925 05:08:01.889640 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:01.889642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:01.889833 (+ 191us) server_negotiation.cc:407] Connection header received
0925 05:08:01.890368 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:01.890370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:01.890382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:01.890435 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:01.891337 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.891821 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.892825 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.893031 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.893130 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.895172 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:01.895174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:01.895174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:01.895181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:01.896501 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:01.896801 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:01.896802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:01.896803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:01.896845 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:01.897058 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:01.897058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:01.897058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:01.897217 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:01.897265 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:01.897488 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 05:08:01.897549 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 05:08:01.897742 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:01.889497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:51321)
0925 05:08:01.889701 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:01.889740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:01.890236 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:01.890504 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:01.890508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:01.891047 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:01.891232 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.891237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.891939 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.891940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.892729 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:01.892732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:01.893141 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:01.893142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:01.893274 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:01.893727 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:01.893735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:01.895086 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:01.896566 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:01.896571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:01.896572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:01.896746 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:01.896899 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:01.896900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:01.896900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:01.897002 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:01.897259 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:01.897263 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:01.897382 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 05:08:01.897462 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 05:08:01.897566 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260925 05:08:01.898257 28521 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40555
I20260925 05:08:01.898345 28521 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:01.898530 28521 heartbeater.cc:511] Master 127.14.33.62:40555 requested a full tablet report, sending...
I20260925 05:08:01.898738 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28392
I20260925 05:08:01.898837 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:08:01.898927 28196 ts_manager.cc:194] Registered new tserver with Master: 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855)
I20260925 05:08:01.899036 14468 external_mini_cluster.cc:1428] Setting key 6e87634a6c52adf4cc963d9b2b5b3660
I20260925 05:08:01.899428 28196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:51321
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:08:01.921252 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40555
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:01.998056 28526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:01.998267 28526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:01.998302 28526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:01.999372 28526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:01.999434 28526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:01.999477 28526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:08:02.000839 28526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:02.001106 28526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:02.001348 28526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:02.005900 28531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.005910 28532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.005913 28534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.006088 28526 server_base.cc:1034] running on GCE node
I20260925 05:08:02.006333 28526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:02.006517 28526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:02.007648 28526 hybrid_clock.cc:648] HybridClock initialized: now 1790312882007632 us; error 31 us; skew 500 ppm
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:02.015085 28526 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:08:02.015493 28526 webserver.cc:460] Webserver started at http://127.14.33.3:34945/ using document root <none> and password file <none>
I20260925 05:08:02.015709 28526 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:02.015751 28526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:02.015897 28526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:02.016835 28526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "412e477edc9d4b289b295108c5e4cdfd"
format_stamp: "Formatted at 2026-09-25 05:08:02 on dist-test-slave-4dmq"
server_key: "ff66a56fa13ff3db79e6f1371d0562bb"
server_key_iv: "804b3ff9874300e9f728b419e4e044ac"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.017220 28526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "412e477edc9d4b289b295108c5e4cdfd"
format_stamp: "Formatted at 2026-09-25 05:08:02 on dist-test-slave-4dmq"
server_key: "ff66a56fa13ff3db79e6f1371d0562bb"
server_key_iv: "804b3ff9874300e9f728b419e4e044ac"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.018393 28526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:02.019058 28541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.019223 28526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:08:02.019323 28526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "412e477edc9d4b289b295108c5e4cdfd"
format_stamp: "Formatted at 2026-09-25 05:08:02 on dist-test-slave-4dmq"
server_key: "ff66a56fa13ff3db79e6f1371d0562bb"
server_key_iv: "804b3ff9874300e9f728b419e4e044ac"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.019405 28526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:02.030735 28526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:02.032231 28526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:02.032369 28526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:02.032547 28526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:02.032821 28526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:02.032872 28526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.032909 28526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:02.032935 28526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.038357 28526 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:46709
I20260925 05:08:02.038394 28654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:46709 every 8 connection(s)
I20260925 05:08:02.038663 28526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:02.045179 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28526
I20260925 05:08:02.045302 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:08:02.045496 14468 external_mini_cluster.cc:1428] Setting key d54c8f458b15d9f153ccdb1d372f4891
I20260925 05:08:02.047533 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.039355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:45969 (local address 127.14.33.62:40555)
0925 05:08:02.039399 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.039401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.039564 (+ 163us) server_negotiation.cc:407] Connection header received
0925 05:08:02.039961 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.039962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.039971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.040021 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.040793 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.041334 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.042227 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.042423 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.042470 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.044339 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.044341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.044341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.044349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.046196 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.046590 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.046591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.046592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.046648 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.046907 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.046908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.046908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.047023 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.047092 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.047362 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.047446 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:08:02.047668 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.039310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:45969)
0925 05:08:02.039443 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.039478 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.039883 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.040065 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.040068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.040561 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.040725 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.040728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.041414 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.041415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.042147 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.042149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.042488 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.042489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.042580 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.042934 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.042941 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.044220 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:02.046282 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.046288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.046291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.046499 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.046734 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.046735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.046735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.046839 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.047094 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:02.047100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:02.047231 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.047327 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.047467 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260925 05:08:02.048066 28655 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40555
I20260925 05:08:02.048163 28655 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:02.048327 28655 heartbeater.cc:511] Master 127.14.33.62:40555 requested a full tablet report, sending...
I20260925 05:08:02.048645 28195 ts_manager.cc:194] Registered new tserver with Master: 412e477edc9d4b289b295108c5e4cdfd (127.14.33.3:46709)
I20260925 05:08:02.049011 28195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:45969
I20260925 05:08:02.057101 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:08:02.063755 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.058373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.14.33.62:40555)
0925 05:08:02.058410 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.058411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.058527 (+ 116us) server_negotiation.cc:407] Connection header received
0925 05:08:02.058562 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.058562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.058570 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.058614 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.059110 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.059538 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.060226 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.060393 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.060438 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.061643 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.061644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.061645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.061651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.062811 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.063145 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.063146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.063147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.063203 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.063361 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.063362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.063362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.063483 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.063565 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.063649 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.063699 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:08:02.065258 28195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:08:02.066691 28195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:08:02.072454 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.068723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:46709 (local address 127.0.0.1:40920)
0925 05:08:02.068975 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.068985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.069023 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.069370 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.069372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.069374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.069449 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.069451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.070328 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.070329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.071405 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.071407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.072141 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.072142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.072233 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.072249 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.072334 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.072384 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260925 05:08:02.072643 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.068914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.14.33.3:46709)
0925 05:08:02.069227 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.069233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.069246 (+ 13us) server_negotiation.cc:407] Connection header received
0925 05:08:02.069277 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.069279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.069290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.069372 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.069521 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.070228 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.071498 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.072059 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.072139 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.072200 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.072352 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.072388 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":64,"threads_started":1}
I20260925 05:08:02.072906 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.068716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:34855 (local address 127.0.0.1:59132)
0925 05:08:02.068880 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.068914 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.068974 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.069693 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.069694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.069696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.069728 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.069730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.070873 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.070874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.071810 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.071813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.072674 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.072675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.072733 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.072736 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.072798 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.072835 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
I20260925 05:08:02.073057 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.068727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32803 (local address 127.0.0.1:56778)
0925 05:08:02.069050 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.069060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.069112 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.069333 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.069336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.069338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.069397 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.069399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.070708 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.070709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.071733 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.071736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.072831 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.072831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.072891 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.072894 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.072940 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.072982 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":43,"threads_started":1}
I20260925 05:08:02.073228 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.068812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.14.33.2:34855)
0925 05:08:02.069490 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.069494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.069502 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:08:02.069537 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.069538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.069553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.069621 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.069811 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.070736 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.071894 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.072527 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.072580 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.072642 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.073069 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.073140 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":86,"threads_started":1}
I20260925 05:08:02.073366 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.068961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.14.33.1:32803)
0925 05:08:02.069154 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.069159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.069170 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:08:02.069225 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.069226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.069246 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.069329 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.069601 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.070606 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.071813 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.072498 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.072719 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.072806 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.073223 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.073273 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260925 05:08:02.073428 28589 tablet_service.cc:1467] Processing CreateTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb (DEFAULT_TABLE table=test-table [id=137659c6fe9b4601aebee2e24133646f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:02.073781 28589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554fe5acf4ca44b9bbeb24eae58103eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:02.073915 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb (DEFAULT_TABLE table=test-table [id=137659c6fe9b4601aebee2e24133646f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:02.074051 28321 tablet_service.cc:1467] Processing CreateTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb (DEFAULT_TABLE table=test-table [id=137659c6fe9b4601aebee2e24133646f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:02.074244 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554fe5acf4ca44b9bbeb24eae58103eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:02.074363 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554fe5acf4ca44b9bbeb24eae58103eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:02.076742 28669 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Bootstrap starting.
I20260925 05:08:02.077433 28670 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Bootstrap starting.
I20260925 05:08:02.077462 28669 tablet_bootstrap.cc:654] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:02.077853 28671 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Bootstrap starting.
I20260925 05:08:02.077983 28669 log.cc:824] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.078145 28670 tablet_bootstrap.cc:654] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:02.078508 28671 tablet_bootstrap.cc:654] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:02.078714 28670 log.cc:824] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.078766 28669 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: No bootstrap required, opened a new log
I20260925 05:08:02.078819 28669 ts_tablet_manager.cc:1397] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:02.079007 28671 log.cc:824] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.079397 28670 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: No bootstrap required, opened a new log
I20260925 05:08:02.079495 28670 ts_tablet_manager.cc:1397] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:08:02.079692 28671 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: No bootstrap required, opened a new log
I20260925 05:08:02.079766 28671 ts_tablet_manager.cc:1397] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:02.080660 28669 raft_consensus.cc:348] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.080823 28669 raft_consensus.cc:374] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:02.080858 28669 raft_consensus.cc:729] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e477edc9d4b289b295108c5e4cdfd, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.081017 28669 consensus_queue.cc:260] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.081315 28669 ts_tablet_manager.cc:1428] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:08:02.081444 28655 heartbeater.cc:503] Master 127.14.33.62:40555 was elected leader, sending a full tablet report...
I20260925 05:08:02.081502 28670 raft_consensus.cc:348] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.081629 28670 raft_consensus.cc:374] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:02.081650 28670 raft_consensus.cc:729] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302cf4b4efbb4b56af459cf52b189bc2, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.081780 28670 consensus_queue.cc:260] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.081928 28671 raft_consensus.cc:348] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.082048 28670 ts_tablet_manager.cc:1428] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:08:02.082048 28521 heartbeater.cc:503] Master 127.14.33.62:40555 was elected leader, sending a full tablet report...
I20260925 05:08:02.082301 28671 raft_consensus.cc:374] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:02.082363 28671 raft_consensus.cc:729] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2456116ca0b427784d3057b261ca566, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.082571 28671 consensus_queue.cc:260] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.082852 28671 ts_tablet_manager.cc:1428] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:08:02.082916 28387 heartbeater.cc:503] Master 127.14.33.62:40555 was elected leader, sending a full tablet report...
W20260925 05:08:02.139456 28522 tablet.cc:2369] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:08:02.150605 28675 raft_consensus.cc:482] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:08:02.150740 28675 raft_consensus.cc:504] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.151139 28675 leader_election.cc:288] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855), f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803)
I20260925 05:08:02.154742 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.151336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:57115 (local address 127.14.33.2:34855)
0925 05:08:02.151380 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.151382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.151391 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:08:02.151423 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.151424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.151434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.151498 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.151681 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.152414 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.153818 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.154382 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.154450 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.154461 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.154624 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.154662 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:08:02.154728 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.151375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:60089 (local address 127.14.33.1:32803)
0925 05:08:02.151419 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.151420 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.151577 (+ 157us) server_negotiation.cc:407] Connection header received
0925 05:08:02.151620 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.151621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.151631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.151703 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.151839 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.152489 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.153818 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.154378 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.154450 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.154461 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.154624 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.154662 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:08:02.154768 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.151265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:34855 (local address 127.14.33.3:57115)
0925 05:08:02.151309 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.151323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.151368 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.151528 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.151529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.151531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.151606 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.151608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.152483 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.152484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.153731 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.153734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.154462 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.154463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.154550 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.154552 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.154603 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.154639 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":9}
I20260925 05:08:02.154778 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.151332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32803 (local address 127.14.33.3:60089)
0925 05:08:02.151508 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.151517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.151565 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.151723 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.151723 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.151724 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.151776 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.151777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.152558 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.152559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.153731 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.153734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.154462 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.154463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.154539 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.154543 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.154599 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.154639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":51,"thread_start_us":106,"threads_started":1}
I20260925 05:08:02.155124 28341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554fe5acf4ca44b9bbeb24eae58103eb" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2456116ca0b427784d3057b261ca566" is_pre_election: true
I20260925 05:08:02.155108 28475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554fe5acf4ca44b9bbeb24eae58103eb" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302cf4b4efbb4b56af459cf52b189bc2" is_pre_election: true
I20260925 05:08:02.155306 28341 raft_consensus.cc:2454] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 0.
I20260925 05:08:02.155304 28475 raft_consensus.cc:2454] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 0.
I20260925 05:08:02.155508 28543 leader_election.cc:302] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e477edc9d4b289b295108c5e4cdfd, f2456116ca0b427784d3057b261ca566; no voters:
I20260925 05:08:02.155643 28675 raft_consensus.cc:2781] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:08:02.155699 28675 raft_consensus.cc:482] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:08:02.155714 28675 raft_consensus.cc:3037] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:02.156484 28675 raft_consensus.cc:504] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.156611 28675 leader_election.cc:288] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 election: Requested vote from peers 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855), f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803)
I20260925 05:08:02.156729 28475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554fe5acf4ca44b9bbeb24eae58103eb" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302cf4b4efbb4b56af459cf52b189bc2"
I20260925 05:08:02.156801 28475 raft_consensus.cc:3037] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:02.156811 28341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554fe5acf4ca44b9bbeb24eae58103eb" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2456116ca0b427784d3057b261ca566"
I20260925 05:08:02.156888 28341 raft_consensus.cc:3037] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:02.157563 28475 raft_consensus.cc:2454] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 1.
I20260925 05:08:02.157676 28341 raft_consensus.cc:2454] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 1.
I20260925 05:08:02.157723 28542 leader_election.cc:302] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302cf4b4efbb4b56af459cf52b189bc2, 412e477edc9d4b289b295108c5e4cdfd; no voters:
I20260925 05:08:02.157827 28675 raft_consensus.cc:2781] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:02.157990 28675 raft_consensus.cc:686] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 1 LEADER]: Becoming Leader. State: Replica: 412e477edc9d4b289b295108c5e4cdfd, State: Running, Role: LEADER
I20260925 05:08:02.158097 28675 consensus_queue.cc:237] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.158943 28196 catalog_manager.cc:5495] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 412e477edc9d4b289b295108c5e4cdfd (127.14.33.3). New cstate: current_term: 1 leader_uuid: "412e477edc9d4b289b295108c5e4cdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } health_report { overall_health: UNKNOWN } } }
I20260925 05:08:02.167340 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.164961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.14.33.3:46709)
0925 05:08:02.165020 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.165022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.165065 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:08:02.165202 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.165203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.165220 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.165297 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:02.165434 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.165873 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.166588 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.166793 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.166865 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.167022 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:02.167119 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:08:02.167214 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.167230 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.167249 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:08:02.168885 28341 raft_consensus.cc:1261] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Refusing update from remote peer 412e477edc9d4b289b295108c5e4cdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:02.168885 28475 raft_consensus.cc:1261] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Refusing update from remote peer 412e477edc9d4b289b295108c5e4cdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:02.169226 28675 consensus_queue.cc:1035] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:02.169368 28679 consensus_queue.cc:1035] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:02.171202 28682 mvcc.cc:205] Tried to move back new op lower bound from 7333121565361401856 to 7333121565320339456. Current Snapshot: MvccSnapshot[applied={T|T < 7333121565361401856}]
I20260925 05:08:02.171960 28683 mvcc.cc:205] Tried to move back new op lower bound from 7333121565361401856 to 7333121565320339456. Current Snapshot: MvccSnapshot[applied={T|T < 7333121565361401856}]
I20260925 05:08:02.173444 28196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:02.173514 28196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:02.174152 28196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Sending DeleteTablet for 3 replicas of tablet 554fe5acf4ca44b9bbeb24eae58103eb
I20260925 05:08:02.174465 28455 tablet_service.cc:1514] Processing DeleteTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:59132
I20260925 05:08:02.174484 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:56778
I20260925 05:08:02.174490 28589 tablet_service.cc:1514] Processing DeleteTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:40920
I20260925 05:08:02.174690 28693 tablet_replica.cc:330] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: stopping tablet replica
I20260925 05:08:02.174759 28693 raft_consensus.cc:2229] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:02.174762 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28258
I20260925 05:08:02.174844 28693 raft_consensus.cc:2258] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.174970 28692 tablet_replica.cc:330] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: stopping tablet replica
I20260925 05:08:02.175038 28692 raft_consensus.cc:2229] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:02.175107 28692 raft_consensus.cc:2258] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.175750 28692 ts_tablet_manager.cc:1905] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:02.177459 28692 ts_tablet_manager.cc:1918] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:02.177522 28692 log.cc:1196] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/554fe5acf4ca44b9bbeb24eae58103eb
I20260925 05:08:02.177773 28692 ts_tablet_manager.cc:1939] T 554fe5acf4ca44b9bbeb24eae58103eb P 302cf4b4efbb4b56af459cf52b189bc2: Deleting consensus metadata
I20260925 05:08:02.178158 28180 catalog_manager.cc:4841] TS 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855): tablet 554fe5acf4ca44b9bbeb24eae58103eb (table test-table [id=137659c6fe9b4601aebee2e24133646f]) successfully deleted
W20260925 05:08:02.180562 28181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:32803 (error 108)
I20260925 05:08:02.180815 28694 tablet_replica.cc:330] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: stopping tablet replica
I20260925 05:08:02.180821 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28392
I20260925 05:08:02.180914 28694 raft_consensus.cc:2229] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:08:02.181017 28694 raft_consensus.cc:2258] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.181407 28694 ts_tablet_manager.cc:1905] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:02.182616 28694 ts_tablet_manager.cc:1918] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:02.182703 28694 log.cc:1196] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/554fe5acf4ca44b9bbeb24eae58103eb
I20260925 05:08:02.182945 28694 ts_tablet_manager.cc:1939] T 554fe5acf4ca44b9bbeb24eae58103eb P 412e477edc9d4b289b295108c5e4cdfd: Deleting consensus metadata
I20260925 05:08:02.183308 28182 catalog_manager.cc:4841] TS 412e477edc9d4b289b295108c5e4cdfd (127.14.33.3:46709): tablet 554fe5acf4ca44b9bbeb24eae58103eb (table test-table [id=137659c6fe9b4601aebee2e24133646f]) successfully deleted
I20260925 05:08:02.188009 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28526
I20260925 05:08:02.189888 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.189759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32803 (local address 127.0.0.1:56792)
0925 05:08:02.189807 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.189853 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:32803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 05:08:02.191066 28181 catalog_manager.cc:4584] TS f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 554fe5acf4ca44b9bbeb24eae58103eb: Network error: Client connection negotiation failed: client connection to 127.14.33.1:32803: connect: Connection refused (error 111)
I20260925 05:08:02.194833 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28164
I20260925 05:08:02.200266 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40555
--webserver_interface=127.14.33.62
--webserver_port=38269
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:02.276324 28696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:02.276533 28696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:02.276572 28696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:02.277704 28696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:02.277762 28696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:02.277786 28696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:02.277805 28696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:02.277833 28696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:02.279211 28696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=38269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:02.279466 28696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:02.279675 28696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:02.287721 28705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.287714 28702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.287914 28696 server_base.cc:1034] running on GCE node
W20260925 05:08:02.287711 28703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.288249 28696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:02.288559 28696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:02.289721 28696 hybrid_clock.cc:648] HybridClock initialized: now 1790312882289708 us; error 40 us; skew 500 ppm
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:02.292137 28696 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:08:02.292600 28696 webserver.cc:460] Webserver started at http://127.14.33.62:38269/ using document root <none> and password file <none>
I20260925 05:08:02.292834 28696 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:02.292909 28696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:02.294327 28696 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:02.295099 28712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.295289 28696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:02.295389 28696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "00d222d8b08445f7a31a13b8de0edb4a"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "bc78719d8c070b20cb112894fc8f2856"
server_key_iv: "218743524cb0c85fed817b2dd5fb4f2a"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.295758 28696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:02.321417 28696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:02.323192 28696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:02.323344 28696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:02.327189 28696 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40555
I20260925 05:08:02.327234 28764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40555 every 8 connection(s)
I20260925 05:08:02.327601 28696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:02.329165 28765 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:08:02.329823 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap starting.
I20260925 05:08:02.334720 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28696
I20260925 05:08:02.334981 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:32803
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:40555
--webserver_port=33441
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:08:02.337847 28765 log.cc:824] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.339545 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:08:02.339850 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap complete.
I20260925 05:08:02.342124 28765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:02.342377 28765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d222d8b08445f7a31a13b8de0edb4a, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.342553 28765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:02.342628 28765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:02.342681 28765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:02.342757 28765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:08:02.343698 28765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:02.343813 28765 leader_election.cc:302] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d222d8b08445f7a31a13b8de0edb4a; no voters:
I20260925 05:08:02.344026 28765 leader_election.cc:288] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:08:02.344059 28769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:08:02.344210 28769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 LEADER]: Becoming Leader. State: Replica: 00d222d8b08445f7a31a13b8de0edb4a, State: Running, Role: LEADER
I20260925 05:08:02.344278 28765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:02.344403 28769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:02.344765 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "00d222d8b08445f7a31a13b8de0edb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } } }
I20260925 05:08:02.344852 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:02.345044 28771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d222d8b08445f7a31a13b8de0edb4a. Latest consensus state: current_term: 2 leader_uuid: "00d222d8b08445f7a31a13b8de0edb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } } }
I20260925 05:08:02.345161 28778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:02.345183 28771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:02.345872 28778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:02.345966 28778 catalog_manager.cc:1259] Loaded cluster ID: e674cb03d23a4d67b1e8b2cb4dd93375
I20260925 05:08:02.345994 28778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:08:02.346274 28778 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 05:08:02.351227 28778 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 05:08:02.353881 28778 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:02.354054 28778 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Loaded TSK: 0
I20260925 05:08:02.354261 28778 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:08:02.416074 28767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:02.416251 28767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:02.416285 28767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:02.417430 28767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:02.417469 28767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:02.417493 28767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:08:02.418928 28767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=33441
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:02.419133 28767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:02.419374 28767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:02.423956 28793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.424028 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.423951 28792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.424228 28767 server_base.cc:1034] running on GCE node
I20260925 05:08:02.424420 28767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:02.424652 28767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:02.425808 28767 hybrid_clock.cc:648] HybridClock initialized: now 1790312882425789 us; error 35 us; skew 500 ppm
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:02.428217 28767 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:08:02.428658 28767 webserver.cc:460] Webserver started at http://127.14.33.1:33441/ using document root <none> and password file <none>
I20260925 05:08:02.428890 28767 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:02.428980 28767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:02.430546 28767 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:02.431206 28802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.431380 28767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:02.431478 28767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "f2456116ca0b427784d3057b261ca566"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "ab4747f3e9df893cab018a0ed4b48c52"
server_key_iv: "1b1be04efde391a85e2f7b827edc7bba"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.431819 28767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:02.451583 28767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:02.453943 28767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:02.454113 28767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:02.454360 28767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:02.454834 28809 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:08:02.456192 28767 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:08:02.456255 28767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:08:02.456293 28767 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:08:02.457193 28767 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:08:02.457242 28767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:02.457290 28809 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Bootstrap starting.
I20260925 05:08:02.464367 28767 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:32803
I20260925 05:08:02.464815 28767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:02.467072 28809 log.cc:824] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.468261 28916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:32803 every 8 connection(s)
I20260925 05:08:02.469568 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28767
I20260925 05:08:02.469846 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:34855
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:40555
--webserver_port=39589
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:08:02.470176 28809 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:08:02.470494 28809 tablet_bootstrap.cc:492] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Bootstrap complete.
I20260925 05:08:02.471650 28809 ts_tablet_manager.cc:1397] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260925 05:08:02.473055 28809 raft_consensus.cc:348] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.473423 28809 raft_consensus.cc:729] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2456116ca0b427784d3057b261ca566, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.473552 28809 consensus_queue.cc:260] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.473803 28809 ts_tablet_manager.cc:1428] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:02.481765 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.465569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:58625)
0925 05:08:02.467358 (+ 1789us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.467404 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.467903 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.468920 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.468924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.469286 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.469592 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.469600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.472565 (+ 2965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.472566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.473843 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.473846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.474802 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.474803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.474947 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.475608 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.475619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.477558 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:02.480014 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.480021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.480023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.480331 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.480645 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.480645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.480646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.480750 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.481253 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:02.481261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:02.481409 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.481495 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.481621 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1743,"thread_start_us":118,"threads_started":1}
I20260925 05:08:02.482059 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.465673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:58625 (local address 127.14.33.62:40555)
0925 05:08:02.468543 (+ 2870us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.468550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.468583 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:08:02.468688 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.468693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.468732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.468883 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.471237 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.472458 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.473992 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.474688 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.474820 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.477696 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.477701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.477701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.477715 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.479914 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.480435 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.480439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.480440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.480567 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.480834 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.480835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.480836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.481171 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.481270 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.481591 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.481722 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2820,"thread_start_us":90,"threads_started":1}
I20260925 05:08:02.482628 28917 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40555
I20260925 05:08:02.482738 28917 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:02.482920 28917 heartbeater.cc:511] Master 127.14.33.62:40555 requested a full tablet report, sending...
I20260925 05:08:02.483561 28729 ts_manager.cc:194] Registered new tserver with Master: f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803)
I20260925 05:08:02.484652 28729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:58625
I20260925 05:08:02.485620 28917 heartbeater.cc:503] Master 127.14.33.62:40555 was elected leader, sending a full tablet report...
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:08:02.491957 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.484121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32803 (local address 127.0.0.1:56802)
0925 05:08:02.484505 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.484538 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.484606 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.484873 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.484878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.485226 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.485287 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.485290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.486187 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.486188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.486768 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.486771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.487197 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.487198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.487269 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.487754 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.487758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.489108 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:02.490737 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.490744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.490746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.491012 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.491247 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.491248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.491248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.491308 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.491590 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:02.491596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:02.491643 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.491747 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.491859 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":80,"threads_started":1}
I20260925 05:08:02.491957 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.484352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.14.33.1:32803)
0925 05:08:02.484506 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.484510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.484603 (+ 93us) server_negotiation.cc:407] Connection header received
0925 05:08:02.484678 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.484680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.484696 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.484817 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.485372 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.486084 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.486863 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.487118 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.487189 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.489245 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.489252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.489253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.489259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.490633 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.491102 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.491104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.491105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.491173 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.491377 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.491378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.491378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.491518 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.491608 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.491782 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.491865 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260925 05:08:02.492545 28833 tablet_service.cc:1514] Processing DeleteTablet for tablet 554fe5acf4ca44b9bbeb24eae58103eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:56802
I20260925 05:08:02.493047 28926 tablet_replica.cc:330] stopping tablet replica
I20260925 05:08:02.493168 28926 raft_consensus.cc:2229] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:02.493232 28926 raft_consensus.cc:2258] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.493525 28926 ts_tablet_manager.cc:1905] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:02.495658 28926 ts_tablet_manager.cc:1918] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:02.495733 28926 log.cc:1196] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/554fe5acf4ca44b9bbeb24eae58103eb
I20260925 05:08:02.495951 28926 ts_tablet_manager.cc:1939] T 554fe5acf4ca44b9bbeb24eae58103eb P f2456116ca0b427784d3057b261ca566: Deleting consensus metadata
I20260925 05:08:02.496389 28714 catalog_manager.cc:4841] TS f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803): tablet 554fe5acf4ca44b9bbeb24eae58103eb (table test-table [id=137659c6fe9b4601aebee2e24133646f]) successfully deleted
W20260925 05:08:02.552553 28922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:02.552744 28922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:02.552771 28922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:02.553817 28922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:02.553848 28922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:02.553895 28922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:08:02.555254 28922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:34855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=39589
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:02.555414 28922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:02.555560 28922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:02.560127 28934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.560192 28922 server_base.cc:1034] running on GCE node
W20260925 05:08:02.560161 28931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.560138 28932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.560526 28922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:02.560806 28922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:02.561966 28922 hybrid_clock.cc:648] HybridClock initialized: now 1790312882561954 us; error 25 us; skew 500 ppm
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:02.564399 28922 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:08:02.564818 28922 webserver.cc:460] Webserver started at http://127.14.33.2:39589/ using document root <none> and password file <none>
I20260925 05:08:02.564982 28922 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:02.565025 28922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:02.566180 28922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:02.566624 28941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.566776 28922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.566862 28922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "302cf4b4efbb4b56af459cf52b189bc2"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "44ad4960467887dee6bc17b101711c4a"
server_key_iv: "85b07ec6a8cb394d46448bda1bb44fc9"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.567222 28922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:02.591380 28922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:02.593065 28922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:02.593266 28922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:02.593495 28922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:02.593792 28922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:02.593861 28922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.593901 28922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:02.593930 28922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.599416 28922 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:34855
I20260925 05:08:02.599478 29054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:34855 every 8 connection(s)
I20260925 05:08:02.599749 28922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:02.605851 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 28922
I20260925 05:08:02.606091 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:46709
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:40555
--webserver_port=34945
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:08:02.608481 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.600392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:46419 (local address 127.14.33.62:40555)
0925 05:08:02.600451 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.600453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.600650 (+ 197us) server_negotiation.cc:407] Connection header received
0925 05:08:02.601160 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.601161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.601174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.601217 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.602110 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.602607 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.603554 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.603776 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.603851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.605924 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.605925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.605925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.605932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.607209 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.607603 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.607604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.607604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.607661 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.607899 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.607900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.607900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.608006 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.608075 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.608317 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.608399 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:08:02.608630 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.600399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:46419)
0925 05:08:02.600535 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.600566 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.601045 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.601283 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.601287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.601825 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.601996 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.602000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.602704 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.602705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.603463 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.603465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.603858 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.603859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.603996 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.604480 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.604488 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.605834 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:02.607311 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.607318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.607320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.607532 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.607722 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.607723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.607723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.607829 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.608066 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:02.608072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:02.608198 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.608298 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.608431 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260925 05:08:02.609177 29055 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40555
I20260925 05:08:02.609283 29055 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:02.609457 29055 heartbeater.cc:511] Master 127.14.33.62:40555 requested a full tablet report, sending...
I20260925 05:08:02.609894 28729 ts_manager.cc:194] Registered new tserver with Master: 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855)
I20260925 05:08:02.610282 28729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:46419
W20260925 05:08:02.684576 29058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:02.684769 29058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:02.684791 29058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:02.685942 29058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:02.685982 29058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:02.686005 29058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:08:02.687294 29058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:46709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=34945
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:02.687505 29058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:02.687697 29058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:08:02.692611 29058 server_base.cc:1034] running on GCE node
W20260925 05:08:02.692586 29064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.692602 29063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:02.692770 29066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:02.693030 29058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:02.693301 29058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:02.694437 29058 hybrid_clock.cc:648] HybridClock initialized: now 1790312882694427 us; error 28 us; skew 500 ppm
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:02.696703 29058 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:08:02.697103 29058 webserver.cc:460] Webserver started at http://127.14.33.3:34945/ using document root <none> and password file <none>
I20260925 05:08:02.697374 29058 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:02.697430 29058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:02.698874 29058 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:02.699429 29073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.699599 29058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:08:02.699685 29058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "412e477edc9d4b289b295108c5e4cdfd"
format_stamp: "Formatted at 2026-09-25 05:08:02 on dist-test-slave-4dmq"
server_key: "ff66a56fa13ff3db79e6f1371d0562bb"
server_key_iv: "804b3ff9874300e9f728b419e4e044ac"
server_key_version: "encryptionkey@0"
I20260925 05:08:02.699993 29058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:02.709411 29058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:02.711047 29058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:02.711196 29058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:02.711403 29058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:02.711673 29058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:02.711705 29058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.711724 29058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:02.711737 29058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:02.716979 29058 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:46709
I20260925 05:08:02.717032 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:46709 every 8 connection(s)
I20260925 05:08:02.717360 29058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:08:02.720741 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29058
Sep 25 05:08:02 dist-test-slave-4dmq krb5kdc[28151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312882, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:02.726749 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.718010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:40469 (local address 127.14.33.62:40555)
0925 05:08:02.718049 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.718051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.718268 (+ 217us) server_negotiation.cc:407] Connection header received
0925 05:08:02.719123 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.719124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.719136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.719190 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.719726 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.720177 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.721412 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.721718 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.721790 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.723846 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.723847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.723847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.723855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.725452 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.725837 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.725838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.725839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.725892 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.726147 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.726148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.726148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.726265 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.726339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.726589 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.726661 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:08:02.726949 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.717991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:40469)
0925 05:08:02.718136 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.718173 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.719039 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.719267 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.719270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.719490 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:02.719660 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.719664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.720258 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.720259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.721336 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.721339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.721807 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.721808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.721899 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.722315 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:02.722324 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:02.723764 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:02.725531 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.725537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.725539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.725745 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.725959 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:02.725960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:02.725961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:02.726081 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:02.726329 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:02.726335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:02.726467 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.726568 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.726713 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260925 05:08:02.727444 29187 heartbeater.cc:347] Connected to a master server at 127.14.33.62:40555
I20260925 05:08:02.727543 29187 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:02.727721 29187 heartbeater.cc:511] Master 127.14.33.62:40555 requested a full tablet report, sending...
I20260925 05:08:02.727833 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.721100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.14.33.62:40555)
0925 05:08:02.721219 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.721221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.721254 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:08:02.721307 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.721308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.721318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.721390 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:02.722048 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.722750 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.723458 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.723758 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.723831 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.725256 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:02.725257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:02.725257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:02.725263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:02.726554 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.726896 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.726896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.726897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.726953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:02.727304 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:02.727305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:02.727305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:02.727427 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:02.727583 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.727653 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.727742 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260925 05:08:02.728073 28729 ts_manager.cc:194] Registered new tserver with Master: 412e477edc9d4b289b295108c5e4cdfd (127.14.33.3:46709)
I20260925 05:08:02.728422 28729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:40469
I20260925 05:08:02.728498 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:08:02.730968 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.728831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.14.33.62:40555)
0925 05:08:02.728867 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.728869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.728993 (+ 124us) server_negotiation.cc:407] Connection header received
0925 05:08:02.729027 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.729028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.729059 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.729167 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:02.729296 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.729726 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.730418 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.730588 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.730635 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.730754 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:02.730830 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:08:02.730881 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.730914 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.730929 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:08:02.731220 28729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:08:02.732646 28729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:08:02.734522 28833 tablet_service.cc:1467] Processing CreateTablet for tablet 58c0892f271f4517813207c4d51401e9 (DEFAULT_TABLE table=test-table [id=1f8ecce686bf4fa79d5f725f1e7e43d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:02.734709 28833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c0892f271f4517813207c4d51401e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:02.736176 28809 tablet_bootstrap.cc:492] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Bootstrap starting.
I20260925 05:08:02.736496 28809 tablet_bootstrap.cc:654] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:02.737040 28809 tablet_bootstrap.cc:492] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: No bootstrap required, opened a new log
I20260925 05:08:02.737104 28809 ts_tablet_manager.cc:1397] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:02.737450 28809 raft_consensus.cc:348] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.737530 28809 raft_consensus.cc:374] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:02.737579 28809 raft_consensus.cc:729] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2456116ca0b427784d3057b261ca566, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.737622 28809 consensus_queue.cc:260] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.737710 28809 ts_tablet_manager.cc:1428] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:02.738493 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.734280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:34855 (local address 127.0.0.1:59140)
0925 05:08:02.734473 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.734482 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.734511 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.734927 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.734928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.734929 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.734975 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.734977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.736293 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.736294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.737347 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.737351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.738249 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.738249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.738321 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.738345 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.738400 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.738432 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":9,"thread_start_us":71,"threads_started":1}
I20260925 05:08:02.738505 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.734480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.14.33.2:34855)
0925 05:08:02.734702 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.734707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.734726 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:08:02.734789 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.734791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.734808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.735082 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.735175 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.736216 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.737449 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.738143 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.738206 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.738270 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.738416 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.738447 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260925 05:08:02.738492 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.734266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:46709 (local address 127.0.0.1:40950)
0925 05:08:02.734386 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.734396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.734442 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.734928 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.734929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.734932 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.734969 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.734970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.736215 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.736216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.737377 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.737381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.738249 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.738250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.738321 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.738343 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.738375 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.738417 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":28}
I20260925 05:08:02.738762 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.734603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.14.33.3:46709)
0925 05:08:02.734776 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.734780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.734787 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:08:02.734819 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.734821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.734836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.734910 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.735049 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.736094 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.737461 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.738156 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.738535 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.738598 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.738623 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.738665 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260925 05:08:02.739142 28989 tablet_service.cc:1467] Processing CreateTablet for tablet 58c0892f271f4517813207c4d51401e9 (DEFAULT_TABLE table=test-table [id=1f8ecce686bf4fa79d5f725f1e7e43d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:02.739387 28989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c0892f271f4517813207c4d51401e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:02.739312 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 58c0892f271f4517813207c4d51401e9 (DEFAULT_TABLE table=test-table [id=1f8ecce686bf4fa79d5f725f1e7e43d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:02.739614 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c0892f271f4517813207c4d51401e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:02.741878 29197 tablet_bootstrap.cc:492] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Bootstrap starting.
I20260925 05:08:02.742185 29198 tablet_bootstrap.cc:492] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Bootstrap starting.
I20260925 05:08:02.742551 29197 tablet_bootstrap.cc:654] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:02.742842 29198 tablet_bootstrap.cc:654] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:02.743063 29197 log.cc:824] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.743355 29198 log.cc:824] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:02.743763 29197 tablet_bootstrap.cc:492] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: No bootstrap required, opened a new log
I20260925 05:08:02.743831 29197 ts_tablet_manager.cc:1397] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:02.744094 29198 tablet_bootstrap.cc:492] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: No bootstrap required, opened a new log
I20260925 05:08:02.744148 29198 ts_tablet_manager.cc:1397] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:02.745587 29197 raft_consensus.cc:348] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.745638 29198 raft_consensus.cc:348] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.745731 29197 raft_consensus.cc:374] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:02.745775 29197 raft_consensus.cc:729] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302cf4b4efbb4b56af459cf52b189bc2, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.745744 29198 raft_consensus.cc:374] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:02.745824 29198 raft_consensus.cc:729] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412e477edc9d4b289b295108c5e4cdfd, State: Initialized, Role: FOLLOWER
I20260925 05:08:02.745889 29197 consensus_queue.cc:260] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.745965 29198 consensus_queue.cc:260] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.746161 29197 ts_tablet_manager.cc:1428] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:08:02.746240 29055 heartbeater.cc:503] Master 127.14.33.62:40555 was elected leader, sending a full tablet report...
I20260925 05:08:02.746237 29198 ts_tablet_manager.cc:1428] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:08:02.746353 29187 heartbeater.cc:503] Master 127.14.33.62:40555 was elected leader, sending a full tablet report...
W20260925 05:08:02.850247 29056 tablet.cc:2369] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:08:02.943923 29202 raft_consensus.cc:482] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:08:02.944075 29202 raft_consensus.cc:504] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.944485 29202 leader_election.cc:288] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855), f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803)
I20260925 05:08:02.948107 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.944738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:35377 (local address 127.14.33.2:34855)
0925 05:08:02.944779 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.944781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.944906 (+ 125us) server_negotiation.cc:407] Connection header received
0925 05:08:02.944933 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.944934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.944946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.945001 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.945151 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.945873 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.947227 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.947726 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.947800 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.947813 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.947990 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.948047 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:08:02.948107 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.944720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:45985 (local address 127.14.33.1:32803)
0925 05:08:02.944760 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.944763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.944795 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:08:02.944822 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.944823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.944849 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.944945 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:02.945164 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.945874 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.947222 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.947781 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.947837 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.947907 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.947981 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.948038 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:08:02.948124 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.944619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:32803 (local address 127.14.33.3:45985)
0925 05:08:02.944678 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.944691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.944748 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.944968 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.944969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.944970 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.945060 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.945062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.945950 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.945951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.947131 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.947134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.947854 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.947855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.947921 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.947923 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.947969 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.948003 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":36}
I20260925 05:08:02.948130 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.944627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:34855 (local address 127.14.33.3:35377)
0925 05:08:02.944818 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:02.944826 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:02.944872 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:02.945010 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:02.945010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:02.945011 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:02.945060 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.945062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.945950 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.945951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.947131 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:02.947134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.947839 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.947840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:02.947909 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.947913 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:08:02.947964 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:08:02.948003 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":92,"thread_start_us":93,"threads_started":1}
I20260925 05:08:02.948443 28871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c0892f271f4517813207c4d51401e9" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2456116ca0b427784d3057b261ca566" is_pre_election: true
I20260925 05:08:02.948475 29009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c0892f271f4517813207c4d51401e9" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302cf4b4efbb4b56af459cf52b189bc2" is_pre_election: true
I20260925 05:08:02.948593 28871 raft_consensus.cc:2454] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 0.
I20260925 05:08:02.948617 29009 raft_consensus.cc:2454] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 0.
I20260925 05:08:02.948817 29074 leader_election.cc:302] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302cf4b4efbb4b56af459cf52b189bc2, 412e477edc9d4b289b295108c5e4cdfd; no voters:
I20260925 05:08:02.949003 29202 raft_consensus.cc:2781] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:08:02.949069 29202 raft_consensus.cc:482] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:08:02.949100 29202 raft_consensus.cc:3037] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:02.949756 29202 raft_consensus.cc:504] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.949894 29202 leader_election.cc:288] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 election: Requested vote from peers 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855), f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803)
I20260925 05:08:02.950033 28871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c0892f271f4517813207c4d51401e9" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2456116ca0b427784d3057b261ca566"
I20260925 05:08:02.950109 28871 raft_consensus.cc:3037] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:02.950083 29009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c0892f271f4517813207c4d51401e9" candidate_uuid: "412e477edc9d4b289b295108c5e4cdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302cf4b4efbb4b56af459cf52b189bc2"
I20260925 05:08:02.950181 29009 raft_consensus.cc:3037] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:02.950780 28871 raft_consensus.cc:2454] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 1.
I20260925 05:08:02.950893 29009 raft_consensus.cc:2454] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412e477edc9d4b289b295108c5e4cdfd in term 1.
I20260925 05:08:02.950929 29075 leader_election.cc:302] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412e477edc9d4b289b295108c5e4cdfd, f2456116ca0b427784d3057b261ca566; no voters:
I20260925 05:08:02.951017 29202 raft_consensus.cc:2781] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:02.951197 29202 raft_consensus.cc:686] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 1 LEADER]: Becoming Leader. State: Replica: 412e477edc9d4b289b295108c5e4cdfd, State: Running, Role: LEADER
I20260925 05:08:02.951347 29202 consensus_queue.cc:237] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } }
I20260925 05:08:02.952186 28728 catalog_manager.cc:5495] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 412e477edc9d4b289b295108c5e4cdfd (127.14.33.3). New cstate: current_term: 1 leader_uuid: "412e477edc9d4b289b295108c5e4cdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412e477edc9d4b289b295108c5e4cdfd" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 } health_report { overall_health: UNKNOWN } } }
I20260925 05:08:02.962998 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:02.960614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.14.33.3:46709)
0925 05:08:02.960671 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:02.960673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:02.960718 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:08:02.960812 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:02.960813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:02.960829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:02.960908 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:02.961015 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.961515 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.962271 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:02.962459 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:02.962536 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:02.962670 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:02.962786 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:08:02.962867 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:02.962891 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 05:08:02.962910 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:08:02.964568 28871 raft_consensus.cc:1261] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Refusing update from remote peer 412e477edc9d4b289b295108c5e4cdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:02.964630 29009 raft_consensus.cc:1261] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Refusing update from remote peer 412e477edc9d4b289b295108c5e4cdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:02.964857 29202 consensus_queue.cc:1035] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2456116ca0b427784d3057b261ca566" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:02.965022 29204 consensus_queue.cc:1035] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "302cf4b4efbb4b56af459cf52b189bc2" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:02.966871 29206 mvcc.cc:205] Tried to move back new op lower bound from 7333121568620716032 to 7333121568569405440. Current Snapshot: MvccSnapshot[applied={T|T < 7333121568620716032}]
I20260925 05:08:02.967170 29207 mvcc.cc:205] Tried to move back new op lower bound from 7333121568620716032 to 7333121568569405440. Current Snapshot: MvccSnapshot[applied={T|T < 7333121568620716032 or (T in {7333121568620716032})}]
I20260925 05:08:02.967808 29209 mvcc.cc:205] Tried to move back new op lower bound from 7333121568620716032 to 7333121568569405440. Current Snapshot: MvccSnapshot[applied={T|T < 7333121568620716032}]
I20260925 05:08:02.969326 28729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:02.969419 28729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:02.970206 28729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Sending DeleteTablet for 3 replicas of tablet 58c0892f271f4517813207c4d51401e9
I20260925 05:08:02.970554 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c0892f271f4517813207c4d51401e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260925 05:08:02.970665 28989 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c0892f271f4517813207c4d51401e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:59140
I20260925 05:08:02.970877 29217 tablet_replica.cc:330] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: stopping tablet replica
I20260925 05:08:02.970980 29217 raft_consensus.cc:2229] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:02.971038 28833 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c0892f271f4517813207c4d51401e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:02 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:56802
I20260925 05:08:02.971088 29217 raft_consensus.cc:2258] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.971127 28926 tablet_replica.cc:330] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: stopping tablet replica
I20260925 05:08:02.971177 28926 raft_consensus.cc:2229] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:02.971231 28926 raft_consensus.cc:2258] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.971534 28926 ts_tablet_manager.cc:1905] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:02.971536 29217 ts_tablet_manager.cc:1905] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:02.973053 28926 ts_tablet_manager.cc:1918] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:02.973055 29217 ts_tablet_manager.cc:1918] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:02.973143 28926 log.cc:1196] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/58c0892f271f4517813207c4d51401e9
I20260925 05:08:02.973182 29217 log.cc:1196] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/58c0892f271f4517813207c4d51401e9
I20260925 05:08:02.973387 28926 ts_tablet_manager.cc:1939] T 58c0892f271f4517813207c4d51401e9 P f2456116ca0b427784d3057b261ca566: Deleting consensus metadata
I20260925 05:08:02.973469 29217 ts_tablet_manager.cc:1939] T 58c0892f271f4517813207c4d51401e9 P 302cf4b4efbb4b56af459cf52b189bc2: Deleting consensus metadata
I20260925 05:08:02.973718 28714 catalog_manager.cc:4841] TS f2456116ca0b427784d3057b261ca566 (127.14.33.1:32803): tablet 58c0892f271f4517813207c4d51401e9 (table test-table [id=1f8ecce686bf4fa79d5f725f1e7e43d4]) successfully deleted
I20260925 05:08:02.973865 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28767
I20260925 05:08:02.973862 28713 catalog_manager.cc:4841] TS 302cf4b4efbb4b56af459cf52b189bc2 (127.14.33.2:34855): tablet 58c0892f271f4517813207c4d51401e9 (table test-table [id=1f8ecce686bf4fa79d5f725f1e7e43d4]) successfully deleted
I20260925 05:08:02.977237 29216 tablet_replica.cc:330] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: stopping tablet replica
I20260925 05:08:02.977351 29216 raft_consensus.cc:2229] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:08:02.977442 29216 raft_consensus.cc:2258] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:02.978376 29216 ts_tablet_manager.cc:1905] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:02.979828 29216 ts_tablet_manager.cc:1918] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:02.979895 29216 log.cc:1196] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/58c0892f271f4517813207c4d51401e9
I20260925 05:08:02.979885 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28922
I20260925 05:08:02.984683 29216 ts_tablet_manager.cc:1939] T 58c0892f271f4517813207c4d51401e9 P 412e477edc9d4b289b295108c5e4cdfd: Deleting consensus metadata
I20260925 05:08:02.985127 28715 catalog_manager.cc:4841] TS 412e477edc9d4b289b295108c5e4cdfd (127.14.33.3:46709): tablet 58c0892f271f4517813207c4d51401e9 (table test-table [id=1f8ecce686bf4fa79d5f725f1e7e43d4]) successfully deleted
I20260925 05:08:02.985641 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 29058
I20260925 05:08:02.989948 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 28696
I20260925 05:08:02.994825 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40555
--webserver_interface=127.14.33.62
--webserver_port=38269
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:03.068727 29218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:03.068919 29218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:03.068939 29218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:03.070037 29218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:03.070076 29218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:03.070091 29218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:03.070102 29218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:03.070152 29218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:03.071544 29218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=38269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:03.071774 29218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:03.072006 29218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:03.076309 29224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:03.076349 29223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.076421 29218 server_base.cc:1034] running on GCE node
W20260925 05:08:03.076360 29226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.076637 29218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:03.076836 29218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:03.077999 29218 hybrid_clock.cc:648] HybridClock initialized: now 1790312883077990 us; error 27 us; skew 500 ppm
Sep 25 05:08:03 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:03.080106 29218 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:08:03.080523 29218 webserver.cc:460] Webserver started at http://127.14.33.62:38269/ using document root <none> and password file <none>
I20260925 05:08:03.080715 29218 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:03.080785 29218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:03.081966 29218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:08:03.082576 29233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.082734 29218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.082844 29218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "00d222d8b08445f7a31a13b8de0edb4a"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "bc78719d8c070b20cb112894fc8f2856"
server_key_iv: "218743524cb0c85fed817b2dd5fb4f2a"
server_key_version: "encryptionkey@0"
I20260925 05:08:03.083160 29218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:03.100556 29218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:03.102196 29218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:03.102344 29218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:03.105907 29218 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40555
I20260925 05:08:03.105938 29285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40555 every 8 connection(s)
I20260925 05:08:03.106281 29218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:03.107630 29286 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:08:03.108141 29286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap starting.
I20260925 05:08:03.108978 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29218
I20260925 05:08:03.109270 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:32803
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:40555
--webserver_port=33441
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:08:03.115780 29286 log.cc:824] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:03.117763 29286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:08:03.118027 29286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a: Bootstrap complete.
I20260925 05:08:03.120082 29286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:03.120288 29286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d222d8b08445f7a31a13b8de0edb4a, State: Initialized, Role: FOLLOWER
I20260925 05:08:03.120450 29286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:03.120518 29286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:03.120563 29286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:03.120608 29286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:08:03.121528 29286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:03.121606 29286 leader_election.cc:302] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d222d8b08445f7a31a13b8de0edb4a; no voters:
I20260925 05:08:03.121810 29286 leader_election.cc:288] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:08:03.121840 29290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:08:03.121954 29290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [term 3 LEADER]: Becoming Leader. State: Replica: 00d222d8b08445f7a31a13b8de0edb4a, State: Running, Role: LEADER
I20260925 05:08:03.122069 29286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:03.122046 29290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } }
I20260925 05:08:03.122315 29290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d222d8b08445f7a31a13b8de0edb4a. Latest consensus state: current_term: 3 leader_uuid: "00d222d8b08445f7a31a13b8de0edb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } } }
I20260925 05:08:03.122371 29290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:03.122539 29291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "00d222d8b08445f7a31a13b8de0edb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d222d8b08445f7a31a13b8de0edb4a" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40555 } } }
I20260925 05:08:03.122649 29294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:03.122684 29291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d222d8b08445f7a31a13b8de0edb4a [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:03.123440 29294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:03.123708 29294 catalog_manager.cc:1259] Loaded cluster ID: e674cb03d23a4d67b1e8b2cb4dd93375
I20260925 05:08:03.123750 29294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 05:08:03.124017 29294 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9f727409ad google::LogMessage::Fail() at ??:0
@ 0x7f9f7274388a google::LogMessage::SendToLog() at ??:0
@ 0x7f9f727404de google::LogMessage::Flush() at ??:0
@ 0x7f9f72741a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9f7a6610f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9f7a66a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9f7460e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9f74603954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9f754676db start_thread at ??:0
@ 0x7f9f7123e61f clone at ??:0
W20260925 05:08:03.219311 29288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:03.219568 29288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:03.219602 29288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:03.221246 29288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:03.221290 29288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:03.221316 29288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:08:03.222718 29288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=33441
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:03.222940 29288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:03.223165 29288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:03.228219 29323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.228262 29288 server_base.cc:1034] running on GCE node
W20260925 05:08:03.228371 29325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:03.228371 29322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.228670 29288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:03.228938 29288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:03.230122 29288 hybrid_clock.cc:648] HybridClock initialized: now 1790312883230084 us; error 54 us; skew 500 ppm
Sep 25 05:08:03 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:03.232529 29288 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:08:03.232933 29288 webserver.cc:460] Webserver started at http://127.14.33.1:33441/ using document root <none> and password file <none>
I20260925 05:08:03.233188 29288 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:03.233251 29288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:03.234459 29288 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:03.235073 29332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.235201 29288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:08:03.235273 29288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "f2456116ca0b427784d3057b261ca566"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "ab4747f3e9df893cab018a0ed4b48c52"
server_key_iv: "1b1be04efde391a85e2f7b827edc7bba"
server_key_version: "encryptionkey@0"
I20260925 05:08:03.235551 29288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:03.254926 29288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:03.257062 29288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:03.257230 29288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:03.257439 29288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:03.257714 29288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:03.257745 29288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.257789 29288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:03.257813 29288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.264261 29288 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:32803
I20260925 05:08:03.264377 29445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:32803 every 8 connection(s)
I20260925 05:08:03.264684 29288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:03.266271 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29288
I20260925 05:08:03.266507 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:34855
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:40555
--webserver_port=39589
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:03.338204 29448 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:08:03.265316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:34675)
0925 05:08:03.265482 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.265527 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:03.266120 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:03.337647 (+ 71527us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
W20260925 05:08:03.338498 29446 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:40555 (0 consecutive failures): Network error: Failed to ping master at 127.14.33.62:40555: Client connection negotiation failed: client connection to 127.14.33.62:40555: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:08:03.338862 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.338734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:40185)
0925 05:08:03.338788 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.338843 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 05:08:03.339254 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.339149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:54669)
0925 05:08:03.339198 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.339221 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 05:08:03.339385 29446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:08:03.347276 29449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:03.347470 29449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:03.347492 29449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:03.348630 29449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:03.348668 29449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:03.348692 29449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:08:03.350121 29449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:34855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=39589
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:03.350346 29449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:03.350561 29449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:03.355144 29455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.355271 29449 server_base.cc:1034] running on GCE node
W20260925 05:08:03.355113 29457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:03.355120 29454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.355533 29449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:03.355816 29449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:03.356971 29449 hybrid_clock.cc:648] HybridClock initialized: now 1790312883356956 us; error 25 us; skew 500 ppm
Sep 25 05:08:03 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:03.359401 29449 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:08:03.359843 29449 webserver.cc:460] Webserver started at http://127.14.33.2:39589/ using document root <none> and password file <none>
I20260925 05:08:03.360069 29449 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:03.360134 29449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:03.361706 29449 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:03.362375 29464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.362553 29449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:03.362650 29449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "302cf4b4efbb4b56af459cf52b189bc2"
format_stamp: "Formatted at 2026-09-25 05:08:01 on dist-test-slave-4dmq"
server_key: "44ad4960467887dee6bc17b101711c4a"
server_key_iv: "85b07ec6a8cb394d46448bda1bb44fc9"
server_key_version: "encryptionkey@0"
I20260925 05:08:03.362972 29449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:03.386919 29449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:03.388656 29449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:03.388806 29449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:03.389042 29449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:03.389351 29449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:03.389405 29449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.389446 29449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:03.389472 29449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.395221 29449 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:34855
I20260925 05:08:03.395267 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:34855 every 8 connection(s)
I20260925 05:08:03.395572 29449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:08:03.396481 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.396150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:50727)
0925 05:08:03.396290 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.396358 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260925 05:08:03.396862 29578 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:40555 (0 consecutive failures): Network error: Failed to ping master at 127.14.33.62:40555: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
I20260925 05:08:03.397132 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.397040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:41081)
0925 05:08:03.397080 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.397105 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 05:08:03.397408 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.397322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:55775)
0925 05:08:03.397365 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.397388 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:08:03.397521 29578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:08:03.401222 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29449
I20260925 05:08:03.401479 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:46709
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:40555
--webserver_port=34945
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:03.476739 29581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:03.476918 29581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:03.476938 29581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:03.478077 29581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:03.478114 29581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:03.478138 29581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:08:03.479425 29581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:46709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=34945
--tserver_master_addrs=127.14.33.62:40555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:03.479584 29581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:03.479730 29581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:03.484165 29589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:03.484150 29586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:03.484138 29587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:03.484436 29581 server_base.cc:1034] running on GCE node
I20260925 05:08:03.484591 29581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:03.484838 29581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:03.485972 29581 hybrid_clock.cc:648] HybridClock initialized: now 1790312883485963 us; error 23 us; skew 500 ppm
Sep 25 05:08:03 dist-test-slave-4dmq krb5kdc[28151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:03.488363 29581 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:08:03.488691 29581 webserver.cc:460] Webserver started at http://127.14.33.3:34945/ using document root <none> and password file <none>
I20260925 05:08:03.488843 29581 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:03.488878 29581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:03.490072 29581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:03.490585 29596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.490746 29581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:03.490845 29581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "412e477edc9d4b289b295108c5e4cdfd"
format_stamp: "Formatted at 2026-09-25 05:08:02 on dist-test-slave-4dmq"
server_key: "ff66a56fa13ff3db79e6f1371d0562bb"
server_key_iv: "804b3ff9874300e9f728b419e4e044ac"
server_key_version: "encryptionkey@0"
I20260925 05:08:03.491140 29581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:03.504915 29581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:03.506572 29581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:03.506707 29581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:03.506898 29581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:03.507156 29581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:03.507210 29581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.507246 29581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:03.507269 29581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:03.512518 29581 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:46709
I20260925 05:08:03.512554 29709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:46709 every 8 connection(s)
I20260925 05:08:03.512879 29581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:08:03.513854 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.513489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:46311)
0925 05:08:03.513624 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.513726 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260925 05:08:03.514220 29710 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:40555 (0 consecutive failures): Network error: Failed to ping master at 127.14.33.62:40555: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
I20260925 05:08:03.514451 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.514375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:34721)
0925 05:08:03.514413 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.514436 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 05:08:03.514683 29712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:03.514621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:51619)
0925 05:08:03.514653 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:03.514671 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 05:08:03.514786 29710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:08:03.515787 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29581
I20260925 05:08:04.339866 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:04.339654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:48389)
0925 05:08:04.339771 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:04.339805 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260925 05:08:04.397964 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:04.397783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:56195)
0925 05:08:04.397896 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:04.397925 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":48,"threads_started":1}
I20260925 05:08:04.515278 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:04.515095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:51479)
0925 05:08:04.515206 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:04.515237 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260925 05:08:05.340442 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:05.340247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:54787)
0925 05:08:05.340361 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:05.340394 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260925 05:08:05.398514 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:05.398335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:44853)
0925 05:08:05.398448 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:05.398480 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260925 05:08:05.515902 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:05.515745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:35533)
0925 05:08:05.515843 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:05.515869 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
I20260925 05:08:06.341028 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:06.340832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:53857)
0925 05:08:06.340951 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:06.340983 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 05:08:06.399118 29721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:06.398925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:43799)
0925 05:08:06.399050 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:06.399081 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260925 05:08:06.516520 29722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:06.516352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:43721)
0925 05:08:06.516455 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:06.516486 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260925 05:08:07.341609 29723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:07.341423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:34787)
0925 05:08:07.341529 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:07.341562 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260925 05:08:07.399672 29724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:07.399495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:38819)
0925 05:08:07.399606 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:07.399637 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260925 05:08:07.517189 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:07.516957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:45005)
0925 05:08:07.517067 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:07.517097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260925 05:08:08.342226 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:08.342017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:53725)
0925 05:08:08.342142 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:08.342176 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260925 05:08:08.400264 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:08.400087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:34305)
0925 05:08:08.400193 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:08.400227 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":50,"threads_started":1}
I20260925 05:08:08.517725 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:08.517543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:34961)
0925 05:08:08.517665 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:08.517694 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
I20260925 05:08:09.342897 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:09.342699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:48527)
0925 05:08:09.342815 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:09.342844 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260925 05:08:09.400848 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:09.400664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:44917)
0925 05:08:09.400783 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:09.400812 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260925 05:08:09.518254 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:09.518093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:46087)
0925 05:08:09.518195 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:09.518220 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260925 05:08:10.343508 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:10.343304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:60603)
0925 05:08:10.343419 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:10.343450 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260925 05:08:10.401474 29733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:10.401294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:42825)
0925 05:08:10.401405 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:10.401437 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260925 05:08:10.518935 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:10.518734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:45161)
0925 05:08:10.518869 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:10.518901 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260925 05:08:11.344156 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:11.343945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:51695)
0925 05:08:11.344075 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:11.344109 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":51,"threads_started":1}
I20260925 05:08:11.402077 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:11.401862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:48263)
0925 05:08:11.401998 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:11.402025 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260925 05:08:11.519536 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:11.519338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:42417)
0925 05:08:11.519472 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:11.519498 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260925 05:08:12.344827 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.344583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:44177)
0925 05:08:12.344742 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:12.344781 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260925 05:08:12.402762 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.402568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:49725)
0925 05:08:12.402697 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:12.402725 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260925 05:08:12.520195 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:12.519983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:44933)
0925 05:08:12.520115 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:12.520163 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260925 05:08:13.345512 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:13.345301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:48045)
0925 05:08:13.345433 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:13.345466 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 05:08:13.403367 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:13.403168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:57019)
0925 05:08:13.403299 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:13.403328 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260925 05:08:13.520779 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:13.520584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:60135)
0925 05:08:13.520701 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:13.520734 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260925 05:08:14.346123 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.345932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:49405)
0925 05:08:14.346044 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:14.346076 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260925 05:08:14.404016 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.403806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:34117)
0925 05:08:14.403945 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:14.403973 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260925 05:08:14.521446 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:14.521257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:36451)
0925 05:08:14.521378 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:14.521409 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260925 05:08:15.346747 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:15.346539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:47597)
0925 05:08:15.346650 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:15.346700 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260925 05:08:15.404588 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:15.404388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:35579)
0925 05:08:15.404518 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:15.404548 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260925 05:08:15.522076 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:15.521880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:43101)
0925 05:08:15.521997 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:15.522033 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260925 05:08:16.347401 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.347212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:52327)
0925 05:08:16.347321 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:16.347355 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260925 05:08:16.405284 29751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.405041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:52011)
0925 05:08:16.405203 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:16.405239 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":58,"threads_started":1}
I20260925 05:08:16.522722 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:16.522512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:44491)
0925 05:08:16.522634 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:16.522681 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260925 05:08:17.348097 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.347884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:60255)
0925 05:08:17.348015 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:17.348049 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260925 05:08:17.405992 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.405806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:35271)
0925 05:08:17.405924 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:17.405953 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260925 05:08:17.523310 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:17.523131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:43731)
0925 05:08:17.523241 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:17.523274 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260925 05:08:18.348989 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:18.348744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.1:57733)
0925 05:08:18.348908 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:18.348943 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260925 05:08:18.406621 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:18.406435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.2:50557)
0925 05:08:18.406543 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:18.406573 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260925 05:08:18.518989 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 29288
I20260925 05:08:18.523541 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 29449
I20260925 05:08:18.525148 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:18.524917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:40555 (local address 127.14.33.3:44385)
0925 05:08:18.525030 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:18.525063 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260925 05:08:18.527736 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 29581
2026-09-25T05:08:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18162 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:08:18 dist-test-slave-4dmq krb5kdc[29761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:08:18 dist-test-slave-4dmq krb5kdc[29761](info): set up 2 sockets
Sep 25 05:08:18 dist-test-slave-4dmq krb5kdc[29761](info): commencing operation
krb5kdc: starting...
W20260925 05:08:19.533999 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:08:19 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:08:19.644093 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38003
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:19.720719 29774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:19.720966 29774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:19.721002 29774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:19.722160 29774 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:08:19.722231 29774 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:08:19.722251 29774 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:08:19.722270 29774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:19.722302 29774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:19.722332 29774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:19.722358 29774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:19.722384 29774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:19.723831 29774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:19.724108 29774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:19.724335 29774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:19.729341 29780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:19.729383 29774 server_base.cc:1034] running on GCE node
W20260925 05:08:19.729362 29782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:19.729342 29779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:19.729856 29774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:19.730086 29774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:19.731222 29774 hybrid_clock.cc:648] HybridClock initialized: now 1790312899731211 us; error 42 us; skew 500 ppm
Sep 25 05:08:19 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:19.733603 29774 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:08:19.733959 29774 webserver.cc:460] Webserver started at http://127.14.33.62:46589/ using document root <none> and password file <none>
I20260925 05:08:19.734130 29774 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:19.734169 29774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:19.734246 29774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:19.735138 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "d4b690fb32f44db681838f8c316d1dbc"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "d87a22018f3d75fe4e745a8c8342083c"
server_key_iv: "a0c147057b47d7acd0719fc97ed915f8"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.735426 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "d4b690fb32f44db681838f8c316d1dbc"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "d87a22018f3d75fe4e745a8c8342083c"
server_key_iv: "a0c147057b47d7acd0719fc97ed915f8"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.736595 29774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:19.737284 29789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:19.737437 29774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:19.737520 29774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "d4b690fb32f44db681838f8c316d1dbc"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "d87a22018f3d75fe4e745a8c8342083c"
server_key_iv: "a0c147057b47d7acd0719fc97ed915f8"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.737574 29774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:19.754000 29774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:19.756170 29774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:19.756291 29774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:19.759994 29774 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38003
I20260925 05:08:19.760056 29841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38003 every 8 connection(s)
I20260925 05:08:19.760388 29774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:19.761248 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:19.763413 29842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap starting.
I20260925 05:08:19.764009 29842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:19.764312 29842 log.cc:824] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:19.764935 29842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: No bootstrap required, opened a new log
I20260925 05:08:19.766482 29842 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:19.766634 29842 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:19.766682 29842 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b690fb32f44db681838f8c316d1dbc, State: Initialized, Role: FOLLOWER
I20260925 05:08:19.766870 29842 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:19.766965 29842 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:19.767006 29842 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:19.767053 29842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:19.767546 29842 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:19.767655 29842 leader_election.cc:302] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b690fb32f44db681838f8c316d1dbc; no voters:
I20260925 05:08:19.767812 29842 leader_election.cc:288] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:08:19.767831 29845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:19.768038 29845 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 LEADER]: Becoming Leader. State: Replica: d4b690fb32f44db681838f8c316d1dbc, State: Running, Role: LEADER
I20260925 05:08:19.768154 29842 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:19.768265 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29774
I20260925 05:08:19.768375 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:08:19.768184 29845 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:19.768599 14468 external_mini_cluster.cc:1428] Setting key f250082ba5175fd4645e70a6a9682216
I20260925 05:08:19.768915 29847 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b690fb32f44db681838f8c316d1dbc. Latest consensus state: current_term: 1 leader_uuid: "d4b690fb32f44db681838f8c316d1dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } } }
I20260925 05:08:19.768992 29847 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:19.769335 29846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b690fb32f44db681838f8c316d1dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } } }
I20260925 05:08:19.769397 29846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:19.769466 29855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:19.770030 29855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:19.771497 29855 catalog_manager.cc:1347] Generated new cluster ID: ed446a907b9f4d239d2b9463569ae244
I20260925 05:08:19.771545 29855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:08:19 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:19.779985 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:19.769259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.14.33.62:38003)
0925 05:08:19.769694 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:19.769703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:19.769899 (+ 196us) server_negotiation.cc:407] Connection header received
0925 05:08:19.770033 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:19.770038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:19.770101 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:19.770228 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:19.770932 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.772243 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.773246 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.773846 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.773937 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:19.776515 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:19.776519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:19.776520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:19.776530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:19.778401 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:19.778763 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:19.778767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:19.778767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:19.778893 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:19.779088 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:19.779089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:19.779090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:19.779395 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:19.779553 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:19.779654 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 05:08:19.779784 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":84,"threads_started":1}
I20260925 05:08:19.783413 29855 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:08:19.783864 29855 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:19.788558 29855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Generated new TSK 0
I20260925 05:08:19.788719 29855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:08:19.815145 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38003
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:19.909705 29869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:19.909885 29869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:19.909906 29869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:19.910989 29869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:19.911026 29869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:19.911051 29869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:08:19.912528 29869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:19.912750 29869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:19.912985 29869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:19.917706 29875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:19.917725 29877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:19.917755 29869 server_base.cc:1034] running on GCE node
W20260925 05:08:19.917729 29874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:19.918195 29869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:19.918426 29869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:19.919610 29869 hybrid_clock.cc:648] HybridClock initialized: now 1790312899919595 us; error 31 us; skew 500 ppm
Sep 25 05:08:19 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:19.922138 29869 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:08:19.922533 29869 webserver.cc:460] Webserver started at http://127.14.33.1:37111/ using document root <none> and password file <none>
I20260925 05:08:19.922758 29869 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:19.922823 29869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:19.922986 29869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:19.923983 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "996ae001de6b42e0a628dce459a534c4"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "436f01e86d0e93fa12410b32a3a5644f"
server_key_iv: "8c3dc3b6f1e81f66045b8b3ac98a22d8"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.924328 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "996ae001de6b42e0a628dce459a534c4"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "436f01e86d0e93fa12410b32a3a5644f"
server_key_iv: "8c3dc3b6f1e81f66045b8b3ac98a22d8"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.925727 29869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:08:19.926393 29884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:19.926578 29869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:19.926679 29869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "996ae001de6b42e0a628dce459a534c4"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "436f01e86d0e93fa12410b32a3a5644f"
server_key_iv: "8c3dc3b6f1e81f66045b8b3ac98a22d8"
server_key_version: "encryptionkey@0"
I20260925 05:08:19.926757 29869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:19.940920 29869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:19.942562 29869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:19.942719 29869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:19.942929 29869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:19.943209 29869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:19.943264 29869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:19.943302 29869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:19.943328 29869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:19.948729 29869 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:44839
I20260925 05:08:19.948812 29997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:44839 every 8 connection(s)
I20260925 05:08:19.949079 29869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:19.949949 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 29869
I20260925 05:08:19.950057 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:08:19.950233 14468 external_mini_cluster.cc:1428] Setting key 69452bc24724b9d0386b2118898f4e65
Sep 25 05:08:19 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
I20260925 05:08:19.958464 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:19.949773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:46187 (local address 127.14.33.62:38003)
0925 05:08:19.949837 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:19.949839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:19.950050 (+ 211us) server_negotiation.cc:407] Connection header received
0925 05:08:19.950660 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:19.950661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:19.950673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:19.950720 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:19.951591 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.952301 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.953322 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.953662 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.953736 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:19.955683 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:19.955685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:19.955686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:19.955694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:19.957252 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:19.957592 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:19.957593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:19.957594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:19.957652 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:19.957868 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:19.957869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:19.957870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:19.958000 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:19.958070 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:19.958277 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 05:08:19.958356 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:08:19.958552 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:19.949781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:46187)
0925 05:08:19.949934 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:19.949972 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:19.950566 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:19.950804 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:19.950808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:19.951328 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:19.951496 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:19.951500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.952394 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.952395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:19.953248 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:19.953251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:19.953749 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:19.953750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:19.953838 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:19.954318 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:19.954325 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:19.955592 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:19.957322 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:19.957327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:19.957329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:19.957518 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:19.957720 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:19.957720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:19.957721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:19.957803 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:19.958063 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:19.958066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:19.958164 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 05:08:19.958235 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:08:19.958335 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260925 05:08:19.959016 29998 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38003
I20260925 05:08:19.959111 29998 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:19.959314 29998 heartbeater.cc:511] Master 127.14.33.62:38003 requested a full tablet report, sending...
I20260925 05:08:19.959780 29806 ts_manager.cc:194] Registered new tserver with Master: 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839)
I20260925 05:08:19.960372 29806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:46187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:08:19.974328 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38003
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:20.052003 30003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:20.052183 30003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:20.052205 30003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:20.053375 30003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:20.053412 30003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:20.053436 30003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:08:20.054744 30003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:20.054975 30003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:20.055178 30003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:20.059836 30009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.059850 30008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.059888 30011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:20.059955 30003 server_base.cc:1034] running on GCE node
I20260925 05:08:20.060220 30003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:20.060460 30003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:20.061622 30003 hybrid_clock.cc:648] HybridClock initialized: now 1790312900061606 us; error 47 us; skew 500 ppm
Sep 25 05:08:20 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:20.063905 30003 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:08:20.064266 30003 webserver.cc:460] Webserver started at http://127.14.33.2:36343/ using document root <none> and password file <none>
I20260925 05:08:20.064441 30003 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:20.064477 30003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:20.064560 30003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:20.065536 30003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "fa19896e72da443585be990de29e0942"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "ff4de36ae91a1963280dd33bee146b36"
server_key_iv: "5b07fa61127e9b77cfa242fabd23085a"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.065824 30003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "fa19896e72da443585be990de29e0942"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "ff4de36ae91a1963280dd33bee146b36"
server_key_iv: "5b07fa61127e9b77cfa242fabd23085a"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.066989 30003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:20.067608 30018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.067771 30003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:20.067844 30003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "fa19896e72da443585be990de29e0942"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "ff4de36ae91a1963280dd33bee146b36"
server_key_iv: "5b07fa61127e9b77cfa242fabd23085a"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.067894 30003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:20.100685 30003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:20.102639 30003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:20.102746 30003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:20.102910 30003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:20.103211 30003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:20.103243 30003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.103264 30003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:20.103310 30003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.108938 30003 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:39879
I20260925 05:08:20.109005 30131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:39879 every 8 connection(s)
I20260925 05:08:20.109326 30003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:08:20 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:20.117276 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.109988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:35817 (local address 127.14.33.62:38003)
0925 05:08:20.110061 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.110064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.110228 (+ 164us) server_negotiation.cc:407] Connection header received
0925 05:08:20.110644 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.110645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.110655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.110711 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:20.111504 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.111982 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.112889 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.113058 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.113107 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.115062 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:20.115063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:20.115063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:20.115069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:20.116187 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.116526 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.116526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.116527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.116573 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.116766 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.116766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.116766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.116855 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:20.116902 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.117105 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.117216 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:08:20.117486 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.109948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:35817)
0925 05:08:20.110110 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.110142 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.110571 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.110751 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.110754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.111265 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:20.111436 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.111441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.112073 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.112074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.112820 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.112822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.113172 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.113173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.113278 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.113750 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:20.113758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:20.114987 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:20.116261 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.116267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.116268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.116463 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.116627 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.116628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.116628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.116715 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.116905 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:20.116909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:20.117007 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.117080 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.117248 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260925 05:08:20.117967 30132 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38003
I20260925 05:08:20.118062 30132 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:20.118257 30132 heartbeater.cc:511] Master 127.14.33.62:38003 requested a full tablet report, sending...
I20260925 05:08:20.118382 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30003
I20260925 05:08:20.118487 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:08:20.118624 29806 ts_manager.cc:194] Registered new tserver with Master: fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
I20260925 05:08:20.118719 14468 external_mini_cluster.cc:1428] Setting key d567c940c33033490227f911c43e411c
I20260925 05:08:20.119047 29806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:35817
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:08:20.141065 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38003
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:20.218822 30137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:20.219002 30137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:20.219022 30137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:20.220024 30137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:20.220057 30137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:20.220081 30137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:08:20.221447 30137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:20.221614 30137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:20.221761 30137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:20.226456 30145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.226495 30142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:20.226454 30143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:20.226809 30137 server_base.cc:1034] running on GCE node
I20260925 05:08:20.227007 30137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:20.227234 30137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:20.228389 30137 hybrid_clock.cc:648] HybridClock initialized: now 1790312900228387 us; error 45 us; skew 500 ppm
Sep 25 05:08:20 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:20.230912 30137 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:08:20.231356 30137 webserver.cc:460] Webserver started at http://127.14.33.3:36015/ using document root <none> and password file <none>
I20260925 05:08:20.231566 30137 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:20.231628 30137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:20.231765 30137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:08:20.232729 30137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "9f90605362874249b5524b9c5f93293c"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "0d79043b419ede74e1702d37633756ce"
server_key_iv: "8bdcee2bac382dc007540320085935db"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.233073 30137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "9f90605362874249b5524b9c5f93293c"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "0d79043b419ede74e1702d37633756ce"
server_key_iv: "8bdcee2bac382dc007540320085935db"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.234388 30137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:20.235074 30152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.235253 30137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:20.235348 30137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "9f90605362874249b5524b9c5f93293c"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "0d79043b419ede74e1702d37633756ce"
server_key_iv: "8bdcee2bac382dc007540320085935db"
server_key_version: "encryptionkey@0"
I20260925 05:08:20.235427 30137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:20.248795 30137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:20.251036 30137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:20.251187 30137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:20.251395 30137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:20.251686 30137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:20.251739 30137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.251777 30137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:20.251803 30137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:20.257711 30137 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:35837
I20260925 05:08:20.257748 30265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:35837 every 8 connection(s)
I20260925 05:08:20.258062 30137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:08:20 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:20.265290 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30137
I20260925 05:08:20.265398 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:08:20.265612 14468 external_mini_cluster.cc:1428] Setting key 27532e116bb4f45ecb5a071d491d7ce4
I20260925 05:08:20.266525 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.258790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:53287 (local address 127.14.33.62:38003)
0925 05:08:20.258832 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.258834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.259034 (+ 200us) server_negotiation.cc:407] Connection header received
0925 05:08:20.259494 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.259495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.259504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.259558 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:20.260402 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.260882 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.261873 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.262058 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.262109 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.263930 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:20.263932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:20.263932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:20.263938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:20.265079 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.265608 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.265609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.265609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.265662 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.265920 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.265921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.265921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.266027 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:20.266093 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.266360 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.266440 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:08:20.266688 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.258751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:53287)
0925 05:08:20.258911 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.258949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.259418 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.259608 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.259611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.260143 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:20.260324 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.260328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.260956 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.260956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.261803 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.261806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.262124 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.262125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.262212 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.262599 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:20.262607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:20.263854 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:20.265213 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.265220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.265222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.265538 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.265732 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:20.265733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:20.265734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:20.265845 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:20.266098 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:20.266103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:20.266230 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.266325 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.266465 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260925 05:08:20.267066 30266 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38003
I20260925 05:08:20.267163 30266 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:20.267325 30266 heartbeater.cc:511] Master 127.14.33.62:38003 requested a full tablet report, sending...
I20260925 05:08:20.267678 29805 ts_manager.cc:194] Registered new tserver with Master: 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837)
I20260925 05:08:20.268050 29805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:53287
I20260925 05:08:20.277163 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:08:20.284483 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.278517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.14.33.62:38003)
0925 05:08:20.278568 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.278570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.278709 (+ 139us) server_negotiation.cc:407] Connection header received
0925 05:08:20.278758 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.278758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.278766 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.278813 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:20.279439 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.279915 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.280643 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.280843 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.280918 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.282088 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:20.282090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:20.282090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:20.282097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:20.283245 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.283630 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.283632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.283632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.283693 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:20.283927 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:20.283927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:20.283928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:20.284071 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:20.284173 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.284303 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.284376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:08:20.285888 29805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:08:20.287388 29805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:08:20.293707 30274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.289538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39879 (local address 127.0.0.1:51706)
0925 05:08:20.289959 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.289968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.290020 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.290277 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.290280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.290282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:20.290355 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.290358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.291749 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.291750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.292549 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.292551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.293414 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.293415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.293489 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.293503 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.293577 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.293625 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":9,"thread_start_us":34,"threads_started":1}
I20260925 05:08:20.293735 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.289742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.14.33.2:39879)
0925 05:08:20.289933 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.289938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.290030 (+ 92us) server_negotiation.cc:407] Connection header received
0925 05:08:20.290097 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.290099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.290114 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.290195 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:20.290446 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.291587 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.292668 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.293328 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.293410 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.293464 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.293605 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.293647 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260925 05:08:20.294384 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.289511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:35837 (local address 127.0.0.1:53076)
0925 05:08:20.289788 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.289802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.289879 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.290369 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.290369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.290371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:20.290415 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.290416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.291790 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.291791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.293225 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.293231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.294140 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.294141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.294205 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.294210 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.294266 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.294306 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":338,"thread_start_us":65,"threads_started":1}
I20260925 05:08:20.294483 30066 tablet_service.cc:1467] Processing CreateTablet for tablet e686a2c51392464bb17e677efc5be434 (DEFAULT_TABLE table=test-table [id=ec1806389e4e471e861a5318f3f77cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:20.294730 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.289924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.14.33.3:35837)
0925 05:08:20.290179 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.290184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.290193 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:08:20.290248 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.290250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.290266 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.290369 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:20.290522 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.291652 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.293305 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.294049 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.294121 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.294172 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.294579 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.294625 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260925 05:08:20.294817 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.289505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44839 (local address 127.0.0.1:46830)
0925 05:08:20.289759 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.289799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.289863 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.290277 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.290280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.290285 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:20.290354 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.290356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.291708 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.291709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.292980 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.292982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.294362 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.294363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.294640 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.294645 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.294702 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.294744 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":34,"thread_start_us":65,"threads_started":1}
I20260925 05:08:20.294881 30066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e686a2c51392464bb17e677efc5be434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:20.295033 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.289631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.14.33.1:44839)
0925 05:08:20.290022 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.290026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.290037 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:08:20.290077 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.290079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.290096 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.290184 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:20.290444 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.291592 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.293071 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.294197 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.294274 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.294328 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.294911 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.294956 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":74,"threads_started":1}
I20260925 05:08:20.295413 30200 tablet_service.cc:1467] Processing CreateTablet for tablet e686a2c51392464bb17e677efc5be434 (DEFAULT_TABLE table=test-table [id=ec1806389e4e471e861a5318f3f77cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:20.295595 29932 tablet_service.cc:1467] Processing CreateTablet for tablet e686a2c51392464bb17e677efc5be434 (DEFAULT_TABLE table=test-table [id=ec1806389e4e471e861a5318f3f77cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:20.295758 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e686a2c51392464bb17e677efc5be434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:20.295897 29932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e686a2c51392464bb17e677efc5be434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:20.297856 30280 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Bootstrap starting.
I20260925 05:08:20.298456 30280 tablet_bootstrap.cc:654] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:20.298956 30280 log.cc:824] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:20.299198 30282 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Bootstrap starting.
I20260925 05:08:20.299324 30281 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Bootstrap starting.
I20260925 05:08:20.299899 30282 tablet_bootstrap.cc:654] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:20.299947 30280 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: No bootstrap required, opened a new log
I20260925 05:08:20.300001 30280 ts_tablet_manager.cc:1397] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:20.300002 30281 tablet_bootstrap.cc:654] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:20.300487 30282 log.cc:824] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:20.300547 30281 log.cc:824] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:20.301311 30282 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: No bootstrap required, opened a new log
I20260925 05:08:20.301309 30281 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: No bootstrap required, opened a new log
I20260925 05:08:20.301388 30281 ts_tablet_manager.cc:1397] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:08:20.301393 30282 ts_tablet_manager.cc:1397] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:08:20.302313 30280 raft_consensus.cc:348] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.302482 30280 raft_consensus.cc:374] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:20.302518 30280 raft_consensus.cc:729] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa19896e72da443585be990de29e0942, State: Initialized, Role: FOLLOWER
I20260925 05:08:20.302651 30280 consensus_queue.cc:260] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.302870 30282 raft_consensus.cc:348] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.303067 30282 raft_consensus.cc:374] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:20.303117 30282 raft_consensus.cc:729] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996ae001de6b42e0a628dce459a534c4, State: Initialized, Role: FOLLOWER
I20260925 05:08:20.303253 30282 consensus_queue.cc:260] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.303332 30281 raft_consensus.cc:348] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.303525 30281 raft_consensus.cc:374] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:20.303565 30281 raft_consensus.cc:729] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f90605362874249b5524b9c5f93293c, State: Initialized, Role: FOLLOWER
I20260925 05:08:20.303583 30282 ts_tablet_manager.cc:1428] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:08:20.303718 30281 consensus_queue.cc:260] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.303766 29998 heartbeater.cc:503] Master 127.14.33.62:38003 was elected leader, sending a full tablet report...
I20260925 05:08:20.303946 30280 ts_tablet_manager.cc:1428] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 05:08:20.304020 30266 heartbeater.cc:503] Master 127.14.33.62:38003 was elected leader, sending a full tablet report...
I20260925 05:08:20.304049 30132 heartbeater.cc:503] Master 127.14.33.62:38003 was elected leader, sending a full tablet report...
I20260925 05:08:20.304018 30281 ts_tablet_manager.cc:1428] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 05:08:20.359850 30133 tablet.cc:2369] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:08:20.449934 29999 tablet.cc:2369] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:08:20.466794 30286 raft_consensus.cc:482] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:08:20.466898 30286 raft_consensus.cc:504] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.467248 30286 leader_election.cc:288] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839), 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837)
I20260925 05:08:20.470448 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.467402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44839 (local address 127.14.33.2:53709)
0925 05:08:20.467457 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.467470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.467517 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.467684 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.467685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.467686 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:20.467758 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.467760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.468590 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.468591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.469564 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.469566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.470212 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.470214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.470286 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.470291 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.470340 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.470383 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:08:20.470499 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.467473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:53709 (local address 127.14.33.1:44839)
0925 05:08:20.467523 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.467526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.467540 (+ 14us) server_negotiation.cc:407] Connection header received
0925 05:08:20.467576 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.467577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.467592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.467670 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:20.467831 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.468493 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.469667 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.470134 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.470186 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.470194 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.470353 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.470388 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 05:08:20.470638 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.467547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:39943 (local address 127.14.33.3:35837)
0925 05:08:20.467591 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.467593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.467713 (+ 120us) server_negotiation.cc:407] Connection header received
0925 05:08:20.467762 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.467763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.467771 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.467841 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:20.468009 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.468666 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.469848 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.470253 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.470318 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.470328 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.470541 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.470581 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:08:20.470765 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.467456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:35837 (local address 127.14.33.2:39943)
0925 05:08:20.467646 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.467653 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:20.467709 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:20.467848 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:20.467849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:20.467850 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:20.467933 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.467935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.468771 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.468772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.469748 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:20.469751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.470342 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.470343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:20.470425 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.470429 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:08:20.470490 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:08:20.470669 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":131,"threads_started":1}
I20260925 05:08:20.470988 29952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e686a2c51392464bb17e677efc5be434" candidate_uuid: "fa19896e72da443585be990de29e0942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996ae001de6b42e0a628dce459a534c4" is_pre_election: true
I20260925 05:08:20.471040 30220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e686a2c51392464bb17e677efc5be434" candidate_uuid: "fa19896e72da443585be990de29e0942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f90605362874249b5524b9c5f93293c" is_pre_election: true
I20260925 05:08:20.471148 29952 raft_consensus.cc:2454] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa19896e72da443585be990de29e0942 in term 0.
I20260925 05:08:20.471148 30220 raft_consensus.cc:2454] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa19896e72da443585be990de29e0942 in term 0.
I20260925 05:08:20.471360 30021 leader_election.cc:302] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942; no voters:
I20260925 05:08:20.471526 30286 raft_consensus.cc:2781] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:08:20.471562 30286 raft_consensus.cc:482] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:08:20.471577 30286 raft_consensus.cc:3037] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:20.472206 30286 raft_consensus.cc:504] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.472335 30286 leader_election.cc:288] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [CANDIDATE]: Term 1 election: Requested vote from peers 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839), 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837)
I20260925 05:08:20.472497 30220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e686a2c51392464bb17e677efc5be434" candidate_uuid: "fa19896e72da443585be990de29e0942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f90605362874249b5524b9c5f93293c"
I20260925 05:08:20.472494 29952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e686a2c51392464bb17e677efc5be434" candidate_uuid: "fa19896e72da443585be990de29e0942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996ae001de6b42e0a628dce459a534c4"
I20260925 05:08:20.472556 30220 raft_consensus.cc:3037] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:20.472595 29952 raft_consensus.cc:3037] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:20.473246 30220 raft_consensus.cc:2454] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa19896e72da443585be990de29e0942 in term 1.
I20260925 05:08:20.473408 30021 leader_election.cc:302] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942; no voters:
I20260925 05:08:20.473431 29952 raft_consensus.cc:2454] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa19896e72da443585be990de29e0942 in term 1.
I20260925 05:08:20.473606 30286 raft_consensus.cc:2781] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:20.473798 30286 raft_consensus.cc:686] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 LEADER]: Becoming Leader. State: Replica: fa19896e72da443585be990de29e0942, State: Running, Role: LEADER
I20260925 05:08:20.473909 30286 consensus_queue.cc:237] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:20.474706 29806 catalog_manager.cc:5495] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa19896e72da443585be990de29e0942 (127.14.33.2). New cstate: current_term: 1 leader_uuid: "fa19896e72da443585be990de29e0942" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } health_report { overall_health: HEALTHY } } }
W20260925 05:08:20.508567 30267 tablet.cc:2369] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:08:20.517611 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.515245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.14.33.2:39879)
0925 05:08:20.515295 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:20.515297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:20.515306 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:08:20.515406 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:20.515407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:20.515425 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:20.515496 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:20.515646 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.516089 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.516826 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:20.517015 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:20.517099 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:20.517294 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:20.517393 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:08:20.517484 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:20.517503 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 05:08:20.517527 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 05:08:20.519309 30220 raft_consensus.cc:1261] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Refusing update from remote peer fa19896e72da443585be990de29e0942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:20.519310 29952 raft_consensus.cc:1261] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Refusing update from remote peer fa19896e72da443585be990de29e0942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:20.519665 30286 consensus_queue.cc:1035] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:20.519790 30290 consensus_queue.cc:1035] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:20.522516 30294 mvcc.cc:205] Tried to move back new op lower bound from 7333121640524550144 to 7333121640341569536. Current Snapshot: MvccSnapshot[applied={T|T < 7333121640524550144 or (T in {7333121640524550144})}]
I20260925 05:08:20.522743 30295 mvcc.cc:205] Tried to move back new op lower bound from 7333121640524550144 to 7333121640341569536. Current Snapshot: MvccSnapshot[applied={T|T < 7333121640524550144}]
I20260925 05:08:20.524801 29805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:20.524896 29805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:20.525630 29805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Sending DeleteTablet for 3 replicas of tablet e686a2c51392464bb17e677efc5be434
I20260925 05:08:20.525964 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 29869
I20260925 05:08:20.525978 29932 tablet_service.cc:1514] Processing DeleteTablet for tablet e686a2c51392464bb17e677efc5be434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:20 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46830
I20260925 05:08:20.526043 30200 tablet_service.cc:1514] Processing DeleteTablet for tablet e686a2c51392464bb17e677efc5be434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:20 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:53076
W20260925 05:08:20.530364 29793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:44839 (error 108)
I20260925 05:08:20.530839 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.530709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44839 (local address 127.0.0.1:46836)
0925 05:08:20.530762 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.530805 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:08:20.531000 29793 catalog_manager.cc:4584] TS 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e686a2c51392464bb17e677efc5be434: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44839: connect: Connection refused (error 111)
I20260925 05:08:20.526295 30066 tablet_service.cc:1514] Processing DeleteTablet for tablet e686a2c51392464bb17e677efc5be434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:20 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:51706
I20260925 05:08:20.531519 30304 tablet_replica.cc:330] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: stopping tablet replica
I20260925 05:08:20.531570 30305 tablet_replica.cc:330] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: stopping tablet replica
I20260925 05:08:20.531599 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30003
I20260925 05:08:20.531682 30305 raft_consensus.cc:2229] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:20.531759 30304 raft_consensus.cc:2229] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:08:20.531787 30305 raft_consensus.cc:2258] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:20.531917 30304 raft_consensus.cc:2258] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:20.532195 30305 ts_tablet_manager.cc:1905] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:20.534132 30305 ts_tablet_manager.cc:1918] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:20.534195 30305 log.cc:1196] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/e686a2c51392464bb17e677efc5be434
I20260925 05:08:20.534459 30305 ts_tablet_manager.cc:1939] T e686a2c51392464bb17e677efc5be434 P 9f90605362874249b5524b9c5f93293c: Deleting consensus metadata
I20260925 05:08:20.534832 29792 catalog_manager.cc:4841] TS 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): tablet e686a2c51392464bb17e677efc5be434 (table test-table [id=ec1806389e4e471e861a5318f3f77cac]) successfully deleted
I20260925 05:08:20.536235 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30137
I20260925 05:08:20.536231 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:20.536122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39879 (local address 127.0.0.1:51712)
0925 05:08:20.536176 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:20.536203 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.2:39879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 05:08:20.540683 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 29774
I20260925 05:08:40.545941 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38003
--webserver_interface=127.14.33.62
--webserver_port=46589
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:08:40.628338 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:40.628553 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:40.628576 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:40.629803 30306 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:08:40.629844 30306 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:08:40.629858 30306 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:08:40.629870 30306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:08:40.629882 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:40.629895 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:40.629943 30306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:08:40.629961 30306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:08:40.631417 30306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:40.631678 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:40.631915 30306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:40.640720 30313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:40.640699 30315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:40.640709 30306 server_base.cc:1034] running on GCE node
W20260925 05:08:40.640741 30312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:40.641275 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:40.641569 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:40.642735 30306 hybrid_clock.cc:648] HybridClock initialized: now 1790312920642723 us; error 30 us; skew 500 ppm
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:40.645344 30306 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:08:40.645752 30306 webserver.cc:460] Webserver started at http://127.14.33.62:46589/ using document root <none> and password file <none>
I20260925 05:08:40.646014 30306 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:40.646100 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:40.647624 30306 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:08:40.648355 30322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:40.648535 30306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:40.648651 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "d4b690fb32f44db681838f8c316d1dbc"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "d87a22018f3d75fe4e745a8c8342083c"
server_key_iv: "a0c147057b47d7acd0719fc97ed915f8"
server_key_version: "encryptionkey@0"
I20260925 05:08:40.649071 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:40.676498 30306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:40.679164 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:40.679337 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:40.683660 30306 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38003
I20260925 05:08:40.683748 30374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38003 every 8 connection(s)
I20260925 05:08:40.684180 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:08:40.685801 30375 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:08:40.686426 30375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap starting.
I20260925 05:08:40.691438 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30306
I20260925 05:08:40.691757 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:44839
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:38003
--webserver_port=37111
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:08:40.695458 30375 log.cc:824] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:40.697464 30375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:08:40.697785 30375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap complete.
I20260925 05:08:40.700032 30375 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:40.700290 30375 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b690fb32f44db681838f8c316d1dbc, State: Initialized, Role: FOLLOWER
I20260925 05:08:40.700500 30375 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:40.700598 30375 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:08:40.700657 30375 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:08:40.700707 30375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 1 FOLLOWER]: Advancing to term 2
I20260925 05:08:40.701778 30375 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:40.701900 30375 leader_election.cc:302] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b690fb32f44db681838f8c316d1dbc; no voters:
I20260925 05:08:40.702096 30375 leader_election.cc:288] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 05:08:40.702128 30379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Leader election won for term 2
I20260925 05:08:40.702252 30379 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 LEADER]: Becoming Leader. State: Replica: d4b690fb32f44db681838f8c316d1dbc, State: Running, Role: LEADER
I20260925 05:08:40.702368 30375 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:08:40.702427 30379 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:08:40.702804 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d4b690fb32f44db681838f8c316d1dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } } }
I20260925 05:08:40.702908 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:40.703217 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b690fb32f44db681838f8c316d1dbc. Latest consensus state: current_term: 2 leader_uuid: "d4b690fb32f44db681838f8c316d1dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } } }
I20260925 05:08:40.703290 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: This master's current role is: LEADER
I20260925 05:08:40.703526 30388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:08:40.704310 30388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:08:40.704537 30388 catalog_manager.cc:1259] Loaded cluster ID: ed446a907b9f4d239d2b9463569ae244
I20260925 05:08:40.704710 30388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:08:40.705973 30388 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:08:40.706146 30388 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Loaded TSK: 0
I20260925 05:08:40.728773 30388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Generated new TSK 1
I20260925 05:08:40.728891 30388 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:08:40.781361 30377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:40.781563 30377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:40.781587 30377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:40.782734 30377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:40.782775 30377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:40.782801 30377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:08:40.784241 30377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:44839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=37111
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:40.784469 30377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:40.784711 30377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:40.789690 30403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:40.789677 30404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:40.789677 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:40.789896 30377 server_base.cc:1034] running on GCE node
I20260925 05:08:40.790107 30377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:40.790349 30377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:40.791520 30377 hybrid_clock.cc:648] HybridClock initialized: now 1790312920791497 us; error 39 us; skew 500 ppm
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:40.794224 30377 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:08:40.794719 30377 webserver.cc:460] Webserver started at http://127.14.33.1:37111/ using document root <none> and password file <none>
I20260925 05:08:40.794943 30377 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:40.795034 30377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:40.796373 30377 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:40.797077 30413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:40.797272 30377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:40.797377 30377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "996ae001de6b42e0a628dce459a534c4"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "436f01e86d0e93fa12410b32a3a5644f"
server_key_iv: "8c3dc3b6f1e81f66045b8b3ac98a22d8"
server_key_version: "encryptionkey@0"
I20260925 05:08:40.797729 30377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:40.816686 30377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:40.818986 30377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:40.819139 30377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:40.819397 30377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:40.819906 30420 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:08:40.821723 30377 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:08:40.821771 30377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:08:40.821806 30377 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:08:40.822466 30377 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:08:40.822510 30377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:08:40.822571 30420 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Bootstrap starting.
I20260925 05:08:40.830299 30377 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:44839
I20260925 05:08:40.830394 30527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:44839 every 8 connection(s)
I20260925 05:08:40.830792 30377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:08:40.833828 30420 log.cc:824] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:40.835901 30420 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:08:40.836210 30420 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Bootstrap complete.
I20260925 05:08:40.837376 30420 ts_tablet_manager.cc:1397] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260925 05:08:40.837399 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30377
I20260925 05:08:40.837661 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:39879
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:38003
--webserver_port=36343
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:40.839118 30420 raft_consensus.cc:348] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:40.839394 30420 raft_consensus.cc:729] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996ae001de6b42e0a628dce459a534c4, State: Initialized, Role: FOLLOWER
I20260925 05:08:40.839602 30420 consensus_queue.cc:260] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:40.839879 30420 ts_tablet_manager.cc:1428] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 05:08:40.842031 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.831533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:34757)
0925 05:08:40.831740 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:40.831791 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:40.832295 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:40.832728 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:40.832731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:40.833014 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:40.833260 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.833265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.834612 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.834613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.835414 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.835416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.836056 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.836057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.836193 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.836797 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:40.836807 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:40.838334 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:40.840396 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.840403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.840405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.840625 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.840889 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.840890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.840890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.840990 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.841421 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:40.841428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:40.841552 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 05:08:40.841663 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 05:08:40.841830 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
I20260925 05:08:40.842027 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.831597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:34757 (local address 127.14.33.62:38003)
0925 05:08:40.831800 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:40.831808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:40.831899 (+ 91us) server_negotiation.cc:407] Connection header received
0925 05:08:40.832516 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:40.832520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:40.832562 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:40.832701 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:40.833379 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.834478 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.835518 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.835913 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.836036 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.838439 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:40.838443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:40.838444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:40.838454 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:40.840301 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.840722 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.840725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.840725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.840817 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.841067 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.841068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.841069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.841347 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:40.841441 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:40.841714 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 05:08:40.841833 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":106,"threads_started":1}
I20260925 05:08:40.842690 30528 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38003
I20260925 05:08:40.842782 30528 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:40.842965 30528 heartbeater.cc:511] Master 127.14.33.62:38003 requested a full tablet report, sending...
I20260925 05:08:40.843668 30339 ts_manager.cc:194] Registered new tserver with Master: 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839)
I20260925 05:08:40.844856 30339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:34757
I20260925 05:08:40.845691 30528 heartbeater.cc:503] Master 127.14.33.62:38003 was elected leader, sending a full tablet report...
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.1@KRBTEST.COM
I20260925 05:08:40.852759 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.844244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44839 (local address 127.0.0.1:46152)
0925 05:08:40.844421 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:40.844470 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:40.844546 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:40.844773 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:40.844780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:40.845196 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:40.845264 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.845267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.846649 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.846650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.847301 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.847303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.847958 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.847959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.848011 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.848499 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:40.848503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:40.849793 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:40.851643 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.851649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.851650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.851837 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.852146 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.852147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.852148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.852205 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.852497 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:40.852500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:40.852539 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:08:40.852606 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:08:40.852692 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260925 05:08:40.852846 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.844473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.14.33.1:44839)
0925 05:08:40.844609 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:40.844613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:40.844623 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:08:40.844652 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:40.844654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:40.844668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:40.844738 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:40.845370 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.846556 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.847383 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.847880 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.847964 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.849870 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:40.849879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:40.849879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:40.849887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:40.851567 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.852018 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.852020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.852021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.852082 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.852266 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.852267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.852268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.852433 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:40.852513 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:40.852659 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 05:08:40.852754 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260925 05:08:40.853199 30447 tablet_service.cc:1514] Processing DeleteTablet for tablet e686a2c51392464bb17e677efc5be434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:20 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46152
I20260925 05:08:40.853701 30537 tablet_replica.cc:330] stopping tablet replica
I20260925 05:08:40.853816 30537 raft_consensus.cc:2229] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:40.853891 30537 raft_consensus.cc:2258] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:40.854168 30537 ts_tablet_manager.cc:1905] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:40.856079 30537 ts_tablet_manager.cc:1918] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:40.856138 30537 log.cc:1196] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal/wals/e686a2c51392464bb17e677efc5be434
I20260925 05:08:40.856360 30537 ts_tablet_manager.cc:1939] T e686a2c51392464bb17e677efc5be434 P 996ae001de6b42e0a628dce459a534c4: Deleting consensus metadata
I20260925 05:08:40.856832 30326 catalog_manager.cc:4841] TS 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839): tablet e686a2c51392464bb17e677efc5be434 (table test-table [id=ec1806389e4e471e861a5318f3f77cac]) successfully deleted
W20260925 05:08:40.922899 30533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:40.923182 30533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:40.923220 30533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:40.924353 30533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:40.924420 30533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:40.924464 30533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:08:40.925978 30533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:39879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=36343
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:40.926254 30533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:40.926476 30533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:40.931349 30542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:40.931412 30543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:40.931375 30533 server_base.cc:1034] running on GCE node
W20260925 05:08:40.931581 30545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:40.931792 30533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:40.932030 30533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:40.933177 30533 hybrid_clock.cc:648] HybridClock initialized: now 1790312920933174 us; error 39 us; skew 500 ppm
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:40.935447 30533 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:08:40.935907 30533 webserver.cc:460] Webserver started at http://127.14.33.2:36343/ using document root <none> and password file <none>
I20260925 05:08:40.936141 30533 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:40.936223 30533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:40.937680 30533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:40.938439 30552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:40.938627 30533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:08:40.938723 30533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "fa19896e72da443585be990de29e0942"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "ff4de36ae91a1963280dd33bee146b36"
server_key_iv: "5b07fa61127e9b77cfa242fabd23085a"
server_key_version: "encryptionkey@0"
I20260925 05:08:40.939069 30533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:40.948066 30533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:40.949903 30533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:40.950084 30533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:40.950301 30533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:40.950700 30559 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:08:40.952234 30533 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:08:40.952304 30533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:08:40.952364 30533 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:08:40.952975 30533 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:08:40.953028 30533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:08:40.953088 30559 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Bootstrap starting.
I20260925 05:08:40.960244 30533 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:39879
I20260925 05:08:40.960379 30666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:39879 every 8 connection(s)
I20260925 05:08:40.960680 30533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:08:40.962405 30559 log.cc:824] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:40.963645 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30533
I20260925 05:08:40.963980 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:35837
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:38003
--webserver_port=36015
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:08:40.965196 30559 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:08:40.965512 30559 tablet_bootstrap.cc:492] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Bootstrap complete.
I20260925 05:08:40.966532 30559 ts_tablet_manager.cc:1397] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:40.967828 30559 raft_consensus.cc:348] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:40.968173 30559 raft_consensus.cc:729] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa19896e72da443585be990de29e0942, State: Initialized, Role: FOLLOWER
I20260925 05:08:40.968318 30559 consensus_queue.cc:260] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:40.968592 30559 ts_tablet_manager.cc:1428] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 05:08:40.971086 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.961482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:50299 (local address 127.14.33.62:38003)
0925 05:08:40.961551 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:40.961554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:40.961822 (+ 268us) server_negotiation.cc:407] Connection header received
0925 05:08:40.962501 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:40.962502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:40.962518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:40.962599 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:40.963416 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.964003 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.965192 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.965462 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.965539 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.967994 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:40.967996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:40.967997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:40.968005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:40.969581 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.970096 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.970097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.970098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.970175 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.970467 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.970467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.970468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.970573 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:40.970622 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:40.970953 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 05:08:40.971024 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:08:40.971481 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.961370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:50299)
0925 05:08:40.961653 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:40.961709 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:40.962372 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:40.962669 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:40.962673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:40.963038 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:40.963308 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.963315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.964175 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.964176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.965028 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.965030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.965573 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.965575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.965692 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.966238 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:40.966246 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:40.967886 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:40.969661 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.969667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.969669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.969998 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.970253 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.970254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.970254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.970398 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.970637 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:40.970643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:40.970814 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 05:08:40.970936 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 05:08:40.971112 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260925 05:08:40.971985 30667 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38003
I20260925 05:08:40.972095 30667 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:40.972257 30667 heartbeater.cc:511] Master 127.14.33.62:38003 requested a full tablet report, sending...
I20260925 05:08:40.972764 30339 ts_manager.cc:194] Registered new tserver with Master: fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
I20260925 05:08:40.973328 30339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:50299
I20260925 05:08:40.974083 30667 heartbeater.cc:503] Master 127.14.33.62:38003 was elected leader, sending a full tablet report...
Sep 25 05:08:40 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.2@KRBTEST.COM
I20260925 05:08:40.980727 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.973063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39879 (local address 127.0.0.1:50706)
0925 05:08:40.973108 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:40.973140 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:40.973183 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:40.973478 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:40.973479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:40.973648 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:40.973694 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.973696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.974890 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.974892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.975754 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:40.975757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.976159 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.976160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:40.976246 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.976588 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:40.976593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:40.977919 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:40.979715 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.979716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.979717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.979882 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.980103 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:40.980104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:40.980105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:40.980164 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:40.980438 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:40.980439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:40.980478 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:08:40.980541 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 05:08:40.980637 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 05:08:40.980739 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:40.973165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.14.33.2:39879)
0925 05:08:40.973300 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:40.973304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:40.973313 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:08:40.973348 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:40.973350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:40.973363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:40.973462 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:40.973772 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.974818 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.975832 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:40.976086 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:40.976134 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:40.978002 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:40.978009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:40.978009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:40.978016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:40.979615 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.979971 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.979974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.979975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.980030 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:40.980239 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:40.980240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:40.980240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:40.980367 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:40.980455 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:40.980577 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 05:08:40.980656 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260925 05:08:40.981078 30601 tablet_service.cc:1514] Processing DeleteTablet for tablet e686a2c51392464bb17e677efc5be434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:20 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:50706
I20260925 05:08:40.981568 30674 tablet_replica.cc:330] stopping tablet replica
I20260925 05:08:40.981675 30674 raft_consensus.cc:2229] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:40.981734 30674 raft_consensus.cc:2258] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:40.981990 30674 ts_tablet_manager.cc:1905] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:40.984088 30674 ts_tablet_manager.cc:1918] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:40.984155 30674 log.cc:1196] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/e686a2c51392464bb17e677efc5be434
I20260925 05:08:40.984365 30674 ts_tablet_manager.cc:1939] T e686a2c51392464bb17e677efc5be434 P fa19896e72da443585be990de29e0942: Deleting consensus metadata
I20260925 05:08:40.985189 30324 catalog_manager.cc:4841] TS fa19896e72da443585be990de29e0942 (127.14.33.2:39879): tablet e686a2c51392464bb17e677efc5be434 (table test-table [id=ec1806389e4e471e861a5318f3f77cac]) successfully deleted
W20260925 05:08:41.049787 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:08:41.050000 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:08:41.050024 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:08:41.051237 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:08:41.051286 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:08:41.051333 30671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:08:41.052799 30671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:35837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=36015
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:08:41.052969 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:08:41.053196 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:08:41.057953 30679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:41.057993 30680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:08:41.057960 30682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:08:41.058336 30671 server_base.cc:1034] running on GCE node
I20260925 05:08:41.058578 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:08:41.058811 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:08:41.059994 30671 hybrid_clock.cc:648] HybridClock initialized: now 1790312921059980 us; error 23 us; skew 500 ppm
Sep 25 05:08:41 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312921, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:08:41.062438 30671 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:08:41.062857 30671 webserver.cc:460] Webserver started at http://127.14.33.3:36015/ using document root <none> and password file <none>
I20260925 05:08:41.063086 30671 fs_manager.cc:359] Metadata directory not provided
I20260925 05:08:41.063170 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:08:41.064484 30671 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:08:41.065039 30689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:41.065219 30671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:08:41.065320 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "9f90605362874249b5524b9c5f93293c"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "0d79043b419ede74e1702d37633756ce"
server_key_iv: "8bdcee2bac382dc007540320085935db"
server_key_version: "encryptionkey@0"
I20260925 05:08:41.065634 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:08:41.075423 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:08:41.077205 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:08:41.077360 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:08:41.077615 30671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:08:41.077906 30671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:08:41.077963 30671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:41.078006 30671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:08:41.078034 30671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:41.084233 30671 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:35837
I20260925 05:08:41.084264 30802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:35837 every 8 connection(s)
I20260925 05:08:41.084591 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:08:41 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312921, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:08:41.091918 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30671
I20260925 05:08:41.094820 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.085314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:50391 (local address 127.14.33.62:38003)
0925 05:08:41.085358 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.085360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.085609 (+ 249us) server_negotiation.cc:407] Connection header received
0925 05:08:41.086073 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.086074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.086086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.086164 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:41.087125 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.087618 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.088595 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.088798 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.088894 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.090958 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:41.090960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:41.090960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:41.090969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:41.093158 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:41.093631 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:41.093632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:41.093633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:41.093694 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:41.094065 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:41.094066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:41.094066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:41.094265 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:41.094366 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.094648 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.094729 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:08:41.094863 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.085321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:50391)
0925 05:08:41.085485 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:41.085534 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:41.085976 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:41.086228 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:41.086232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:41.086819 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:08:41.087008 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.087013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.087712 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.087713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.088496 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.088498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.088903 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.088904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.089022 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.089538 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:08:41.089546 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:08:41.090835 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:08:41.093261 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:41.093266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:41.093269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:41.093534 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:41.093784 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:08:41.093785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:08:41.093785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:08:41.093968 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:08:41.094360 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:08:41.094366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:08:41.094519 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 05:08:41.094603 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 05:08:41.094719 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260925 05:08:41.095355 30803 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38003
I20260925 05:08:41.095466 30803 heartbeater.cc:464] Registering TS with master...
I20260925 05:08:41.095695 30803 heartbeater.cc:511] Master 127.14.33.62:38003 requested a full tablet report, sending...
I20260925 05:08:41.096086 30339 ts_manager.cc:194] Registered new tserver with Master: 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837)
I20260925 05:08:41.096426 30339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:50391
I20260925 05:08:41.100658 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.092505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.14.33.62:38003)
0925 05:08:41.092653 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.092655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.093369 (+ 714us) server_negotiation.cc:407] Connection header received
0925 05:08:41.093469 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.093470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.093483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.093584 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:41.094406 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.095536 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.096529 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.096777 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.096865 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.098165 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:41.098167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:41.098167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:41.098174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:41.099390 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:41.099824 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:41.099825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:41.099825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:41.099888 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:41.100107 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:41.100108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:41.100109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:41.100225 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:41.100395 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.100451 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.100555 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260925 05:08:41.101394 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:08:41.103976 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.101691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.14.33.62:38003)
0925 05:08:41.101750 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.101751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.101809 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:08:41.101856 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.101856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.101880 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.101936 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:41.102009 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.102542 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.103276 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.103542 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.103615 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.103746 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:41.103780 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 05:08:41.103919 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57010: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 05:08:41.104030 30807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57010: token expired
W20260925 05:08:41.104072 30272 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.33.62:38003)
I20260925 05:08:41.109185 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.104283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.14.33.62:38003)
0925 05:08:41.104337 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.104338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.104347 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:08:41.104406 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.104407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.104415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.104466 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:08:41.104910 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.105374 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.105992 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.106164 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.106225 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.107118 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:08:41.107119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:08:41.107120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:08:41.107125 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:08:41.108183 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:41.108500 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:41.108501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:41.108502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:41.108559 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:08:41.108738 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:08:41.108739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:08:41.108739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:08:41.108833 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:08:41.108926 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.109019 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.109082 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:08:41.110181 30339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:08:41.111631 30339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:08:41.113677 30601 tablet_service.cc:1467] Processing CreateTablet for tablet d46d53459e0d43bd9b3ef7c641c44e10 (DEFAULT_TABLE table=test-table [id=c096057fceec456e99a4432ecb7b5ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:41.113698 30447 tablet_service.cc:1467] Processing CreateTablet for tablet d46d53459e0d43bd9b3ef7c641c44e10 (DEFAULT_TABLE table=test-table [id=c096057fceec456e99a4432ecb7b5ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:41.113878 30447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46d53459e0d43bd9b3ef7c641c44e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:41.113878 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46d53459e0d43bd9b3ef7c641c44e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:41.115233 30420 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Bootstrap starting.
I20260925 05:08:41.115335 30559 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Bootstrap starting.
I20260925 05:08:41.115551 30420 tablet_bootstrap.cc:654] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:41.115588 30559 tablet_bootstrap.cc:654] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:41.116112 30420 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: No bootstrap required, opened a new log
I20260925 05:08:41.116163 30420 ts_tablet_manager.cc:1397] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:08:41.116464 30420 raft_consensus.cc:348] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.116545 30420 raft_consensus.cc:374] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:41.116600 30420 raft_consensus.cc:729] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996ae001de6b42e0a628dce459a534c4, State: Initialized, Role: FOLLOWER
I20260925 05:08:41.116652 30420 consensus_queue.cc:260] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.116752 30420 ts_tablet_manager.cc:1428] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 05:08:41.117069 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.113410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:35837 (local address 127.0.0.1:55406)
0925 05:08:41.113453 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:41.113463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:41.113499 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:41.113834 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:41.113835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:41.113837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:41.113869 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.113871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.115099 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.115100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.115957 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.115960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.116837 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.116838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.116895 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.116911 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 05:08:41.116956 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 05:08:41.117002 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 05:08:41.117323 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.113480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.14.33.3:35837)
0925 05:08:41.113656 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.113661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.113671 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:08:41.113707 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.113709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.113724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.113939 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:41.114038 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.115000 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.116058 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.116755 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.116826 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.116878 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.117195 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.117236 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260925 05:08:41.117352 30559 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: No bootstrap required, opened a new log
I20260925 05:08:41.117391 30559 ts_tablet_manager.cc:1397] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:41.117516 30559 raft_consensus.cc:348] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.117561 30559 raft_consensus.cc:374] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:41.117587 30559 raft_consensus.cc:729] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa19896e72da443585be990de29e0942, State: Initialized, Role: FOLLOWER
I20260925 05:08:41.117654 30559 consensus_queue.cc:260] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.117735 30559 ts_tablet_manager.cc:1428] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 05:08:41.118017 30737 tablet_service.cc:1467] Processing CreateTablet for tablet d46d53459e0d43bd9b3ef7c641c44e10 (DEFAULT_TABLE table=test-table [id=c096057fceec456e99a4432ecb7b5ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:08:41.118340 30737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46d53459e0d43bd9b3ef7c641c44e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:08:41.120797 30812 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Bootstrap starting.
I20260925 05:08:41.121482 30812 tablet_bootstrap.cc:654] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Neither blocks nor log segments found. Creating new log.
I20260925 05:08:41.121991 30812 log.cc:824] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:08:41.122583 30812 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: No bootstrap required, opened a new log
I20260925 05:08:41.122634 30812 ts_tablet_manager.cc:1397] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:08:41.123831 30812 raft_consensus.cc:348] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.123953 30812 raft_consensus.cc:374] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:08:41.123974 30812 raft_consensus.cc:729] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f90605362874249b5524b9c5f93293c, State: Initialized, Role: FOLLOWER
I20260925 05:08:41.124080 30812 consensus_queue.cc:260] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.124294 30812 ts_tablet_manager.cc:1428] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:08:41.124337 30803 heartbeater.cc:503] Master 127.14.33.62:38003 was elected leader, sending a full tablet report...
I20260925 05:08:41.141490 30814 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:08:41.141580 30814 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.141938 30814 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
I20260925 05:08:41.144642 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.142075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44839 (local address 127.14.33.3:52023)
0925 05:08:41.142124 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:41.142135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:41.142179 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:41.142360 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:41.142361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:41.142363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:41.142428 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.142429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.143142 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.143143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.143858 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.143860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.144413 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.144414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.144483 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.144487 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:08:41.144543 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:08:41.144579 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 05:08:41.144652 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.142137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:52023 (local address 127.14.33.1:44839)
0925 05:08:41.142186 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.142188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.142220 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:08:41.142244 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.142245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.142272 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.142356 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:41.142497 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.143064 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.143961 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.144334 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.144415 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.144478 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.144549 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.144595 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:08:41.144881 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46d53459e0d43bd9b3ef7c641c44e10" candidate_uuid: "9f90605362874249b5524b9c5f93293c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996ae001de6b42e0a628dce459a534c4" is_pre_election: true
I20260925 05:08:41.144987 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.142162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:41521 (local address 127.14.33.2:39879)
0925 05:08:41.142218 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.142220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.142347 (+ 127us) server_negotiation.cc:407] Connection header received
0925 05:08:41.142466 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.142467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.142485 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.142568 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:08:41.142729 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.143230 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.144162 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.144549 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.144630 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.144688 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.144890 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.144930 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 05:08:41.144991 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.142118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39879 (local address 127.14.33.3:41521)
0925 05:08:41.142280 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:41.142287 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:08:41.142350 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:08:41.142582 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:08:41.142583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:08:41.142585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:08:41.142660 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.142661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.143314 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.143315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.144070 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:08:41.144072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.144631 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.144632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:08:41.144799 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.144805 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 05:08:41.144877 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 05:08:41.144912 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260925 05:08:41.145007 30482 raft_consensus.cc:2454] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f90605362874249b5524b9c5f93293c in term 0.
I20260925 05:08:41.145236 30693 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4, 9f90605362874249b5524b9c5f93293c; no voters:
I20260925 05:08:41.145363 30814 raft_consensus.cc:2781] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:08:41.145430 30814 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:08:41.145347 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46d53459e0d43bd9b3ef7c641c44e10" candidate_uuid: "9f90605362874249b5524b9c5f93293c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa19896e72da443585be990de29e0942" is_pre_election: true
I20260925 05:08:41.145449 30814 raft_consensus.cc:3037] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:41.145498 30621 raft_consensus.cc:2454] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f90605362874249b5524b9c5f93293c in term 0.
I20260925 05:08:41.146081 30814 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.146217 30814 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [CANDIDATE]: Term 1 election: Requested vote from peers 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
I20260925 05:08:41.146348 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46d53459e0d43bd9b3ef7c641c44e10" candidate_uuid: "9f90605362874249b5524b9c5f93293c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996ae001de6b42e0a628dce459a534c4"
I20260925 05:08:41.146365 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46d53459e0d43bd9b3ef7c641c44e10" candidate_uuid: "9f90605362874249b5524b9c5f93293c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa19896e72da443585be990de29e0942"
I20260925 05:08:41.146430 30482 raft_consensus.cc:3037] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:41.146461 30621 raft_consensus.cc:3037] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:08:41.147018 30482 raft_consensus.cc:2454] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f90605362874249b5524b9c5f93293c in term 1.
I20260925 05:08:41.147171 30693 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4, 9f90605362874249b5524b9c5f93293c; no voters:
I20260925 05:08:41.147190 30621 raft_consensus.cc:2454] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f90605362874249b5524b9c5f93293c in term 1.
I20260925 05:08:41.147264 30814 raft_consensus.cc:2781] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:08:41.147403 30814 raft_consensus.cc:686] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 1 LEADER]: Becoming Leader. State: Replica: 9f90605362874249b5524b9c5f93293c, State: Running, Role: LEADER
I20260925 05:08:41.147491 30814 consensus_queue.cc:237] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:08:41.148372 30339 catalog_manager.cc:5495] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f90605362874249b5524b9c5f93293c (127.14.33.3). New cstate: current_term: 1 leader_uuid: "9f90605362874249b5524b9c5f93293c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } health_report { overall_health: UNKNOWN } } }
I20260925 05:08:41.162518 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.160471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.14.33.3:35837)
0925 05:08:41.160544 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:08:41.160546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:08:41.160552 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:08:41.160580 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:08:41.160581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:08:41.160594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:08:41.160647 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:08:41.160776 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.161283 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.161936 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:08:41.162113 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:08:41.162156 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:08:41.162285 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:08:41.162364 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:08:41.162418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:08:41.162447 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 05:08:41.162460 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:08:41.163986 30621 raft_consensus.cc:1261] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Refusing update from remote peer 9f90605362874249b5524b9c5f93293c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:41.163987 30482 raft_consensus.cc:1261] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Refusing update from remote peer 9f90605362874249b5524b9c5f93293c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:08:41.164278 30814 consensus_queue.cc:1035] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:41.164412 30816 consensus_queue.cc:1035] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [LEADER]: Connected to new peer: Peer: permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:08:41.166646 30821 mvcc.cc:205] Tried to move back new op lower bound from 7333121725085462528 to 7333121725020823552. Current Snapshot: MvccSnapshot[applied={T|T < 7333121725085462528}]
I20260925 05:08:41.166640 30820 mvcc.cc:205] Tried to move back new op lower bound from 7333121725085462528 to 7333121725020823552. Current Snapshot: MvccSnapshot[applied={T|T < 7333121725085462528}]
I20260925 05:08:41.168488 30338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:41.168582 30338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:08:41.169366 30338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Sending DeleteTablet for 3 replicas of tablet d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:08:41.169642 30601 tablet_service.cc:1514] Processing DeleteTablet for tablet d46d53459e0d43bd9b3ef7c641c44e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:41 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:50706
I20260925 05:08:41.169641 30737 tablet_service.cc:1514] Processing DeleteTablet for tablet d46d53459e0d43bd9b3ef7c641c44e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:41 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:55406
I20260925 05:08:41.169723 30674 tablet_replica.cc:330] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: stopping tablet replica
I20260925 05:08:41.169772 30674 raft_consensus.cc:2229] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:41.169806 30447 tablet_service.cc:1514] Processing DeleteTablet for tablet d46d53459e0d43bd9b3ef7c641c44e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:08:41 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:46152
I20260925 05:08:41.169838 30674 raft_consensus.cc:2258] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:41.169858 30828 tablet_replica.cc:330] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: stopping tablet replica
I20260925 05:08:41.169884 30537 tablet_replica.cc:330] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: stopping tablet replica
I20260925 05:08:41.169924 30537 raft_consensus.cc:2229] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:08:41.169936 30828 raft_consensus.cc:2229] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:08:41.169971 30537 raft_consensus.cc:2258] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:41.170004 30828 raft_consensus.cc:2258] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:08:41.170217 30674 ts_tablet_manager.cc:1905] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:41.170298 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30377
I20260925 05:08:41.171026 30828 ts_tablet_manager.cc:1905] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:08:41.171809 30674 ts_tablet_manager.cc:1918] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:41.171845 30674 log.cc:1196] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:08:41.172000 30674 ts_tablet_manager.cc:1939] T d46d53459e0d43bd9b3ef7c641c44e10 P fa19896e72da443585be990de29e0942: Deleting consensus metadata
I20260925 05:08:41.172415 30324 catalog_manager.cc:4841] TS fa19896e72da443585be990de29e0942 (127.14.33.2:39879): tablet d46d53459e0d43bd9b3ef7c641c44e10 (table test-table [id=c096057fceec456e99a4432ecb7b5ce8]) successfully deleted
I20260925 05:08:41.172663 30828 ts_tablet_manager.cc:1918] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:08:41.172731 30828 log.cc:1196] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:08:41.173010 30828 ts_tablet_manager.cc:1939] T d46d53459e0d43bd9b3ef7c641c44e10 P 9f90605362874249b5524b9c5f93293c: Deleting consensus metadata
I20260925 05:08:41.174052 30325 catalog_manager.cc:4841] TS 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): tablet d46d53459e0d43bd9b3ef7c641c44e10 (table test-table [id=c096057fceec456e99a4432ecb7b5ce8]) successfully deleted
W20260925 05:08:41.176920 30326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:44839 (error 108)
I20260925 05:08:41.177202 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30533
I20260925 05:08:41.177412 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:08:41.177256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:44839 (local address 127.0.0.1:46168)
0925 05:08:41.177338 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 05:08:41.177377 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:08:41.181365 30326 catalog_manager.cc:4584] TS 996ae001de6b42e0a628dce459a534c4 (127.14.33.1:44839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d46d53459e0d43bd9b3ef7c641c44e10: Network error: Client connection negotiation failed: client connection to 127.14.33.1:44839: connect: Connection refused (error 111)
I20260925 05:08:41.184405 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30671
I20260925 05:08:41.188868 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30306
I20260925 05:09:01.194149 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38003
--webserver_interface=127.14.33.62
--webserver_port=46589
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:01.273545 30830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:01.273730 30830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:01.273751 30830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:01.274891 30830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 05:09:01.274931 30830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 05:09:01.274945 30830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 05:09:01.274991 30830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:01.275009 30830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:01.275033 30830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:01.275059 30830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:01.275086 30830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:01.276552 30830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:01.276800 30830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:01.277053 30830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:01.282157 30838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:01.282192 30836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:01.282207 30830 server_base.cc:1034] running on GCE node
W20260925 05:09:01.282176 30835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:01.282601 30830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:01.282892 30830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:01.284032 30830 hybrid_clock.cc:648] HybridClock initialized: now 1790312941284033 us; error 40 us; skew 500 ppm
Sep 25 05:09:01 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:01.286509 30830 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:01.286936 30830 webserver.cc:460] Webserver started at http://127.14.33.62:46589/ using document root <none> and password file <none>
I20260925 05:09:01.287153 30830 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:01.287195 30830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:01.288450 30830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:01.289086 30845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.289279 30830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:01.289373 30830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "d4b690fb32f44db681838f8c316d1dbc"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "d87a22018f3d75fe4e745a8c8342083c"
server_key_iv: "a0c147057b47d7acd0719fc97ed915f8"
server_key_version: "encryptionkey@0"
I20260925 05:09:01.289701 30830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:01.302739 30830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:01.304412 30830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:01.304550 30830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:01.308501 30830 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38003
I20260925 05:09:01.308534 30897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38003 every 8 connection(s)
I20260925 05:09:01.308903 30830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:01.310240 30898 sys_catalog.cc:263] Verifying existing consensus state
I20260925 05:09:01.310983 30898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap starting.
I20260925 05:09:01.319000 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30830
I20260925 05:09:01.319214 30898 log.cc:824] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:01.319293 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:44839
--local_ip_for_outbound_sockets=127.14.33.1
--tserver_master_addrs=127.14.33.62:38003
--webserver_port=37111
--webserver_interface=127.14.33.1
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:09:01.321453 30898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:09:01.321799 30898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Bootstrap complete.
I20260925 05:09:01.325583 30898 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:09:01.325978 30898 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b690fb32f44db681838f8c316d1dbc, State: Initialized, Role: FOLLOWER
I20260925 05:09:01.326157 30898 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:09:01.326251 30898 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:01.326296 30898 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:01.326349 30898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 2 FOLLOWER]: Advancing to term 3
I20260925 05:09:01.327852 30898 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:09:01.327970 30898 leader_election.cc:302] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b690fb32f44db681838f8c316d1dbc; no voters:
I20260925 05:09:01.328130 30898 leader_election.cc:288] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 05:09:01.328189 30902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 3 FOLLOWER]: Leader election won for term 3
I20260925 05:09:01.328375 30902 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [term 3 LEADER]: Becoming Leader. State: Replica: d4b690fb32f44db681838f8c316d1dbc, State: Running, Role: LEADER
I20260925 05:09:01.328385 30898 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:01.328519 30902 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } }
I20260925 05:09:01.328820 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d4b690fb32f44db681838f8c316d1dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } } }
I20260925 05:09:01.328825 30904 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b690fb32f44db681838f8c316d1dbc. Latest consensus state: current_term: 3 leader_uuid: "d4b690fb32f44db681838f8c316d1dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b690fb32f44db681838f8c316d1dbc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38003 } } }
I20260925 05:09:01.328905 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:01.328913 30904 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:01.329425 30907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:01.330217 30907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:01.330314 30907 catalog_manager.cc:1259] Loaded cluster ID: ed446a907b9f4d239d2b9463569ae244
I20260925 05:09:01.330360 30907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:01.331197 30907 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:01.331322 30907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Loaded TSK: 0
I20260925 05:09:01.331350 30907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4b690fb32f44db681838f8c316d1dbc: Loaded TSK: 1
F20260925 05:09:01.331511 30907 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f126d29d9ad google::LogMessage::Fail() at ??:0
@ 0x7f126d2a088a google::LogMessage::SendToLog() at ??:0
@ 0x7f126d29d4de google::LogMessage::Flush() at ??:0
@ 0x7f126d29ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f126fa0a104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f126fa07d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f12751f71cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f12751f748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f12751f7594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f12751bdc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f12751c7ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f126f16b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f126f160954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f126ffc46db start_thread at ??:0
@ 0x7f126bd9b61f clone at ??:0
W20260925 05:09:01.417656 30901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:01.417834 30901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:01.417855 30901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:01.418939 30901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:01.418978 30901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:01.419026 30901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:01.420526 30901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:44839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=37111
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:01.420750 30901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:01.420969 30901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:01.425789 30939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:01.425828 30942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:01.426003 30901 server_base.cc:1034] running on GCE node
W20260925 05:09:01.425858 30940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:01.426225 30901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:01.426455 30901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:01.427632 30901 hybrid_clock.cc:648] HybridClock initialized: now 1790312941427642 us; error 50 us; skew 500 ppm
Sep 25 05:09:01 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:01.430117 30901 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:09:01.430594 30901 webserver.cc:460] Webserver started at http://127.14.33.1:37111/ using document root <none> and password file <none>
I20260925 05:09:01.430814 30901 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:01.430877 30901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:01.432169 30901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:01.432880 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.433053 30901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:01.433161 30901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "996ae001de6b42e0a628dce459a534c4"
format_stamp: "Formatted at 2026-09-25 05:08:19 on dist-test-slave-4dmq"
server_key: "436f01e86d0e93fa12410b32a3a5644f"
server_key_iv: "8c3dc3b6f1e81f66045b8b3ac98a22d8"
server_key_version: "encryptionkey@0"
I20260925 05:09:01.433475 30901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:01.447348 30901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:01.449435 30901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:01.449584 30901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:01.449780 30901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:01.450181 30956 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 05:09:01.451969 30901 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 05:09:01.452020 30901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 05:09:01.452072 30901 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 05:09:01.452621 30901 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 05:09:01.452669 30901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 05:09:01.452730 30956 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Bootstrap starting.
I20260925 05:09:01.459527 30901 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:44839
I20260925 05:09:01.459622 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:44839 every 8 connection(s)
I20260925 05:09:01.459911 30901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:01.463708 30956 log.cc:824] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:01.464865 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 30901
I20260925 05:09:01.465174 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:39879
--local_ip_for_outbound_sockets=127.14.33.2
--tserver_master_addrs=127.14.33.62:38003
--webserver_port=36343
--webserver_interface=127.14.33.2
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:09:01.466256 30956 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 05:09:01.466563 30956 tablet_bootstrap.cc:492] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Bootstrap complete.
I20260925 05:09:01.467746 30956 ts_tablet_manager.cc:1397] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.009s
I20260925 05:09:01.469101 30956 raft_consensus.cc:348] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:01.469440 30956 raft_consensus.cc:729] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996ae001de6b42e0a628dce459a534c4, State: Initialized, Role: FOLLOWER
I20260925 05:09:01.469607 30956 consensus_queue.cc:260] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:01.469892 30956 ts_tablet_manager.cc:1428] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260925 05:09:01.522601 31066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:01.460886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:53107)
0925 05:09:01.461051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.461093 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:01.461562 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:01.522408 (+ 60846us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
W20260925 05:09:01.522866 31064 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:38003 (0 consecutive failures): Network error: Failed to ping master at 127.14.33.62:38003: Client connection negotiation failed: client connection to 127.14.33.62:38003: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:09:01.523231 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.523101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:43147)
0925 05:09:01.523158 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.523208 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:09:01.523660 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.523552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:33615)
0925 05:09:01.523614 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.523640 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:09:01.523803 31064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 05:09:01.545185 31068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:01.545373 31068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:01.545395 31068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:01.546533 31068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:01.546571 31068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:01.546595 31068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:01.547894 31068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:39879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=36343
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:01.548110 31068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:01.548307 31068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:01.552927 31075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:01.552937 31077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:01.552944 31074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:01.553184 31068 server_base.cc:1034] running on GCE node
I20260925 05:09:01.553406 31068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:01.553592 31068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:01.554721 31068 hybrid_clock.cc:648] HybridClock initialized: now 1790312941554712 us; error 27 us; skew 500 ppm
Sep 25 05:09:01 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:01.557013 31068 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:09:01.557435 31068 webserver.cc:460] Webserver started at http://127.14.33.2:36343/ using document root <none> and password file <none>
I20260925 05:09:01.557662 31068 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:01.557705 31068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:01.559365 31068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:01.560002 31084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.560179 31068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:01.560267 31068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "fa19896e72da443585be990de29e0942"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "ff4de36ae91a1963280dd33bee146b36"
server_key_iv: "5b07fa61127e9b77cfa242fabd23085a"
server_key_version: "encryptionkey@0"
I20260925 05:09:01.560592 31068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:01.576450 31068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:01.578758 31068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:01.578918 31068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:01.579152 31068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:01.579449 31068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:01.579510 31068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.579552 31068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:01.579581 31068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.585399 31068 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:39879
I20260925 05:09:01.585466 31197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:39879 every 8 connection(s)
I20260925 05:09:01.585852 31068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:01.586897 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.586497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:47489)
0925 05:09:01.586655 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.586761 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
W20260925 05:09:01.587289 31198 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:38003 (0 consecutive failures): Network error: Failed to ping master at 127.14.33.62:38003: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
I20260925 05:09:01.587558 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.587460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:56813)
0925 05:09:01.587513 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.587539 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:09:01.587782 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.587733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:45163)
0925 05:09:01.587752 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.587770 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260925 05:09:01.587890 31198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:09:01.590546 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31068
I20260925 05:09:01.590770 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:35837
--local_ip_for_outbound_sockets=127.14.33.3
--tserver_master_addrs=127.14.33.62:38003
--webserver_port=36015
--webserver_interface=127.14.33.3
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:01.669266 31201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:01.669456 31201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:01.669478 31201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:01.670527 31201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:01.670558 31201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:01.670583 31201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:01.671939 31201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:35837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=36015
--tserver_master_addrs=127.14.33.62:38003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:01.672112 31201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:01.672267 31201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:09:01.677066 31201 server_base.cc:1034] running on GCE node
W20260925 05:09:01.677037 31207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:01.677068 31206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:01.677037 31209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:01.677470 31201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:01.677726 31201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:01.678860 31201 hybrid_clock.cc:648] HybridClock initialized: now 1790312941678848 us; error 30 us; skew 500 ppm
Sep 25 05:09:01 dist-test-slave-4dmq krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:01.681195 31201 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:09:01.681599 31201 webserver.cc:460] Webserver started at http://127.14.33.3:36015/ using document root <none> and password file <none>
I20260925 05:09:01.681813 31201 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:01.681856 31201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:01.683094 31201 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:01.683641 31216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.683794 31201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.683887 31201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "9f90605362874249b5524b9c5f93293c"
format_stamp: "Formatted at 2026-09-25 05:08:20 on dist-test-slave-4dmq"
server_key: "0d79043b419ede74e1702d37633756ce"
server_key_iv: "8bdcee2bac382dc007540320085935db"
server_key_version: "encryptionkey@0"
I20260925 05:09:01.684209 31201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:01.694736 31201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:01.696560 31201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:01.696727 31201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:01.696959 31201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:01.697309 31201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:01.697360 31201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.697422 31201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:01.697454 31201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:01.703014 31201 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:35837
I20260925 05:09:01.703068 31329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:35837 every 8 connection(s)
I20260925 05:09:01.703363 31201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:09:01.704267 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.703941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:55565)
0925 05:09:01.704079 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.704142 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20260925 05:09:01.704653 31330 heartbeater.cc:644] Failed to heartbeat to 127.14.33.62:38003 (0 consecutive failures): Network error: Failed to ping master at 127.14.33.62:38003: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
I20260925 05:09:01.704879 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.704816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:58739)
0925 05:09:01.704842 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.704865 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 05:09:01.705205 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:01.705099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:43425)
0925 05:09:01.705151 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:01.705180 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 05:09:01.705257 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31201
W20260925 05:09:01.705353 31330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 05:09:02.524329 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.524119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:50175)
0925 05:09:02.524245 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:02.524272 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260925 05:09:02.588423 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.588236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:33553)
0925 05:09:02.588346 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:02.588379 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260925 05:09:02.705902 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.705656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:35143)
0925 05:09:02.705778 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:02.705836 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260925 05:09:02.996253 31337 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:02.996372 31337 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:02.996754 31337 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
Sep 25 05:09:03 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.3@KRBTEST.COM
Sep 25 05:09:03 dist-test-slave-4dmq krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.2@KRBTEST.COM
I20260925 05:09:03.011920 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.996977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:35837 (local address 127.14.33.1:49503)
0925 05:09:02.997092 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:02.997101 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:02.997174 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:02.998010 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:02.998011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:02.998395 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:02.998571 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:02.998577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:02.999604 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:02.999605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:03.000338 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:03.000340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:03.000899 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:03.000900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:03.000995 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:03.006892 (+ 5897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:03.006900 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:03.008450 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:03.010292 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:03.010298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:03.010300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:03.010577 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:03.010807 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:03.010808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:03.010809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:03.010933 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:03.011368 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:03.011375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:03.011511 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 05:09:03.011620 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 05:09:03.011790 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":60,"thread_start_us":61,"threads_started":1}
I20260925 05:09:03.011965 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.997060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49503 (local address 127.14.33.3:35837)
0925 05:09:02.997242 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:02.997256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:02.997281 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:09:02.997789 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:02.997792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:02.997857 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:02.997994 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:02.998676 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:02.999506 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:03.000454 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:03.000806 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:03.000913 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:03.008590 (+ 7677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:03.008595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:03.008595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:03.008607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:03.010211 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:03.010655 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:03.010658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:03.010658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:03.010746 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:03.011002 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:03.011003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:03.011004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:03.011302 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:03.011371 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:03.011694 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 05:09:03.011837 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260925 05:09:03.012061 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.997052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:42637 (local address 127.14.33.2:39879)
0925 05:09:02.997190 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:02.997204 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:02.997237 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:09:02.997744 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:02.997748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:02.997805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:02.997959 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:02.998674 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:02.999497 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:03.000436 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:03.000795 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:03.000899 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:03.008832 (+ 7933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:03.008842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:03.008842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:03.008871 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:03.010754 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:03.011076 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:03.011078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:03.011078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:03.011147 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:03.011320 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:03.011321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:03.011321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:03.011591 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:03.011660 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:03.011828 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 05:09:03.011948 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260925 05:09:03.011991 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:02.996930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:39879 (local address 127.14.33.1:42637)
0925 05:09:02.996981 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:02.996991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:02.997043 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:02.998004 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:02.998007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:02.998394 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:02.998571 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:02.998577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:02.999597 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:02.999598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:03.000339 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:03.000340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:03.000885 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:03.000886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:03.000993 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:03.006896 (+ 5903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:03.006900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:03.008725 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:03.010833 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:03.010834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:03.010834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:03.011006 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:03.011206 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:03.011207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:03.011207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:03.011263 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:03.011655 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:03.011656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:03.011696 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 05:09:03.011763 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 05:09:03.011844 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":9}
W20260925 05:09:03.013022 30952 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
W20260925 05:09:03.013100 30951 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa19896e72da443585be990de29e0942 (127.14.33.2:39879): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:09:03.013178 30951 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4; no voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942
I20260925 05:09:03.013340 31337 raft_consensus.cc:2726] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:09:03.524906 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:03.524739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:45581)
0925 05:09:03.524835 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:03.524865 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260925 05:09:03.588984 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:03.588807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:52187)
0925 05:09:03.588914 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:03.588945 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260925 05:09:03.706524 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:03.706320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:54929)
0925 05:09:03.706462 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:03.706492 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260925 05:09:04.525638 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:04.525450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:33193)
0925 05:09:04.525562 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:04.525593 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260925 05:09:04.589567 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:04.589355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:44427)
0925 05:09:04.589501 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:04.589533 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 05:09:04.707181 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:04.706913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:59253)
0925 05:09:04.707089 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:04.707144 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260925 05:09:04.848886 31347 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:04.848968 31347 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:04.849208 31347 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
W20260925 05:09:04.849589 30951 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa19896e72da443585be990de29e0942 (127.14.33.2:39879): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
W20260925 05:09:04.849684 30952 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:09:04.849757 30952 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4; no voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942
I20260925 05:09:04.849828 31347 raft_consensus.cc:2726] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:09:05.526270 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:05.526033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:45565)
0925 05:09:05.526189 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:05.526225 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260925 05:09:05.590152 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:05.589973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:52681)
0925 05:09:05.590075 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:05.590116 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260925 05:09:05.707830 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:05.707584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:54235)
0925 05:09:05.707727 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:05.707763 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260925 05:09:06.527043 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:06.526758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:47069)
0925 05:09:06.526963 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:06.526999 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":55,"threads_started":1}
I20260925 05:09:06.590837 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:06.590672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:56221)
0925 05:09:06.590774 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:06.590803 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260925 05:09:06.708426 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:06.708244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:59471)
0925 05:09:06.708362 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:06.708390 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260925 05:09:06.800482 31354 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:06.800621 31354 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:06.800822 31354 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
W20260925 05:09:06.801208 30952 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
W20260925 05:09:06.801311 30951 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa19896e72da443585be990de29e0942 (127.14.33.2:39879): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:09:06.801344 30951 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4; no voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942
I20260925 05:09:06.801432 31354 raft_consensus.cc:2726] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:09:07.527681 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:07.527496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:48177)
0925 05:09:07.527608 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:07.527639 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260925 05:09:07.591492 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:07.591324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:58455)
0925 05:09:07.591428 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:07.591459 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260925 05:09:07.709043 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:07.708856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:35205)
0925 05:09:07.708975 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:07.709006 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260925 05:09:08.528349 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:08.528152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:41651)
0925 05:09:08.528268 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:08.528302 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260925 05:09:08.592185 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:08.591957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:54799)
0925 05:09:08.592111 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:08.592146 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260925 05:09:08.709601 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:08.709417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:57237)
0925 05:09:08.709531 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:08.709562 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260925 05:09:09.528944 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.528743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:41987)
0925 05:09:09.528868 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:09.528899 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260925 05:09:09.592808 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.592609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:36151)
0925 05:09:09.592739 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:09.592770 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260925 05:09:09.710192 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:09.709999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:57617)
0925 05:09:09.710120 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:09.710157 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260925 05:09:10.529608 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.529388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:38817)
0925 05:09:10.529520 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.529557 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260925 05:09:10.593426 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.593233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:60009)
0925 05:09:10.593347 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.593384 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260925 05:09:10.710783 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:10.710576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:41703)
0925 05:09:10.710708 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:10.710743 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260925 05:09:11.094280 31367 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:11.094368 31367 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:11.094527 31367 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
W20260925 05:09:11.094939 30951 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa19896e72da443585be990de29e0942 (127.14.33.2:39879): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
W20260925 05:09:11.095031 30952 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:09:11.095093 30952 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4; no voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942
I20260925 05:09:11.095162 31367 raft_consensus.cc:2726] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:09:11.530314 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:11.530112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:38389)
0925 05:09:11.530237 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:11.530270 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260925 05:09:11.594091 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:11.593868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:48837)
0925 05:09:11.593997 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:11.594043 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260925 05:09:11.711517 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:11.711321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:34509)
0925 05:09:11.711436 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:11.711473 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260925 05:09:12.530975 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:12.530765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:44025)
0925 05:09:12.530901 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:12.530929 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260925 05:09:12.594722 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:12.594490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:59117)
0925 05:09:12.594641 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:12.594677 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260925 05:09:12.712114 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:12.711919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:35037)
0925 05:09:12.712040 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:12.712072 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260925 05:09:13.531572 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:13.531367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:34297)
0925 05:09:13.531491 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:13.531526 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260925 05:09:13.595335 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:13.595128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:46771)
0925 05:09:13.595260 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:13.595292 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260925 05:09:13.712700 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:13.712521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:58131)
0925 05:09:13.712636 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:13.712664 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260925 05:09:14.532228 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:14.532026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:39959)
0925 05:09:14.532150 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:14.532184 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260925 05:09:14.596095 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:14.595867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:42169)
0925 05:09:14.596016 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:14.596052 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 05:09:14.713411 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:14.713145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:33349)
0925 05:09:14.713343 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:14.713375 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":63,"threads_started":1}
I20260925 05:09:15.532943 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:15.532745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:40627)
0925 05:09:15.532865 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:15.532898 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260925 05:09:15.596833 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:15.596618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:50395)
0925 05:09:15.596736 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:15.596774 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260925 05:09:15.714003 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:15.713829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:52165)
0925 05:09:15.713934 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:15.713967 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260925 05:09:16.232853 31383 raft_consensus.cc:482] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:16.232955 31383 raft_consensus.cc:504] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f90605362874249b5524b9c5f93293c" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 35837 } } peers { permanent_uuid: "996ae001de6b42e0a628dce459a534c4" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 44839 } } peers { permanent_uuid: "fa19896e72da443585be990de29e0942" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 39879 } }
I20260925 05:09:16.233136 31383 leader_election.cc:288] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837), fa19896e72da443585be990de29e0942 (127.14.33.2:39879)
W20260925 05:09:16.233500 30952 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f90605362874249b5524b9c5f93293c (127.14.33.3:35837): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
W20260925 05:09:16.233574 30951 leader_election.cc:341] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa19896e72da443585be990de29e0942 (127.14.33.2:39879): Not found: Tablet not found: d46d53459e0d43bd9b3ef7c641c44e10
I20260925 05:09:16.233630 30951 leader_election.cc:302] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 996ae001de6b42e0a628dce459a534c4; no voters: 9f90605362874249b5524b9c5f93293c, fa19896e72da443585be990de29e0942
I20260925 05:09:16.233692 31383 raft_consensus.cc:2726] T d46d53459e0d43bd9b3ef7c641c44e10 P 996ae001de6b42e0a628dce459a534c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 05:09:16.533737 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:16.533520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.1:44177)
0925 05:09:16.533663 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:16.533695 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260925 05:09:16.597612 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:16.597411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.2:48001)
0925 05:09:16.597536 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:16.597570 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260925 05:09:16.712621 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 30901
I20260925 05:09:16.714684 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:16.714495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38003 (local address 127.14.33.3:43971)
0925 05:09:16.714618 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:16.714649 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.62:38003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260925 05:09:16.717423 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31068
I20260925 05:09:16.722213 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31201
2026-09-25T05:09:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58195 ms)
[----------] 30 tests from SecurityITest (227147 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:16 dist-test-slave-4dmq krb5kdc[31389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:16 dist-test-slave-4dmq krb5kdc[31389](info): set up 2 sockets
Sep 25 05:09:16 dist-test-slave-4dmq krb5kdc[31389](info): commencing operation
krb5kdc: starting...
W20260925 05:09:17.741746 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:17 dist-test-slave-4dmq krb5kdc[31389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:17.848029 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36937
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:36937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:17.926483 31402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:17.926677 31402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:17.926698 31402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:17.927784 31402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:17.927820 31402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:17.927834 31402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:17.927846 31402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:17.927858 31402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:17.929270 31402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:36937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:17.929435 31402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:17.929606 31402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:17.934340 31407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:17.934347 31408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:17.934536 31402 server_base.cc:1034] running on GCE node
W20260925 05:09:17.934358 31410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:17.934860 31402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:17.935099 31402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:17.936275 31402 hybrid_clock.cc:648] HybridClock initialized: now 1790312957936255 us; error 40 us; skew 500 ppm
Sep 25 05:09:17 dist-test-slave-4dmq krb5kdc[31389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312957, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:17.938865 31402 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:17.939260 31402 webserver.cc:460] Webserver started at http://127.14.33.62:37877/ using document root <none> and password file <none>
I20260925 05:09:17.939465 31402 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:17.939529 31402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:17.939666 31402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:17.940647 31402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36"
format_stamp: "Formatted at 2026-09-25 05:09:17 on dist-test-slave-4dmq"
server_key: "0969d6a3e4f6e430d546edb74dbb9dfd"
server_key_iv: "c3b4e54405584a87c1fab44f05533ff4"
server_key_version: "encryptionkey@0"
I20260925 05:09:17.941005 31402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36"
format_stamp: "Formatted at 2026-09-25 05:09:17 on dist-test-slave-4dmq"
server_key: "0969d6a3e4f6e430d546edb74dbb9dfd"
server_key_iv: "c3b4e54405584a87c1fab44f05533ff4"
server_key_version: "encryptionkey@0"
I20260925 05:09:17.942458 31402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:17.943230 31417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:17.943411 31402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:09:17.943513 31402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36"
format_stamp: "Formatted at 2026-09-25 05:09:17 on dist-test-slave-4dmq"
server_key: "0969d6a3e4f6e430d546edb74dbb9dfd"
server_key_iv: "c3b4e54405584a87c1fab44f05533ff4"
server_key_version: "encryptionkey@0"
I20260925 05:09:17.943593 31402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:17.956338 31402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:17.958086 31402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:17.958231 31402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:17.962026 31402 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:36937
I20260925 05:09:17.962078 31469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:36937 every 8 connection(s)
I20260925 05:09:17.962514 31402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:17.963400 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:17.965777 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36: Bootstrap starting.
I20260925 05:09:17.966336 31470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:17.966604 31470 log.cc:824] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:17.967231 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36: No bootstrap required, opened a new log
I20260925 05:09:17.968725 31470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36937 } }
I20260925 05:09:17.968899 31470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:17.968947 31470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c87c9be4e3d4da2a1193dfbfe8e3d36, State: Initialized, Role: FOLLOWER
I20260925 05:09:17.969112 31470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36937 } }
I20260925 05:09:17.969226 31470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:17.969269 31470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:17.969318 31470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:17.969812 31470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36937 } }
I20260925 05:09:17.969921 31470 leader_election.cc:302] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c87c9be4e3d4da2a1193dfbfe8e3d36; no voters:
I20260925 05:09:17.970101 31470 leader_election.cc:288] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:17.970162 31473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:17.970348 31470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:17.970357 31473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [term 1 LEADER]: Becoming Leader. State: Replica: 6c87c9be4e3d4da2a1193dfbfe8e3d36, State: Running, Role: LEADER
I20260925 05:09:17.970465 31473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36937 } }
I20260925 05:09:17.970933 31474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36937 } } }
I20260925 05:09:17.971186 31474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:17.971112 31475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c87c9be4e3d4da2a1193dfbfe8e3d36. Latest consensus state: current_term: 1 leader_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c87c9be4e3d4da2a1193dfbfe8e3d36" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36937 } } }
I20260925 05:09:17.971393 31475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:17.971454 31483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:17.971922 31483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:17.972229 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31402
I20260925 05:09:17.972348 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:17.972517 14468 external_mini_cluster.cc:1428] Setting key 2343fc89cedcce1aff6cc79d6791b7d7
I20260925 05:09:17.974170 31483 catalog_manager.cc:1347] Generated new cluster ID: b0419107967c42f2b900d3b2eda6159a
I20260925 05:09:17.974215 31483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:09:17 dist-test-slave-4dmq krb5kdc[31389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:17.983727 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:17.973065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.14.33.62:36937)
0925 05:09:17.973327 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:17.973336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:17.973366 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:09:17.973479 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:17.973486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:17.973537 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:17.973674 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:17.974431 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:17.975831 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:17.976800 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:17.977445 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:17.977551 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:17.980124 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:17.980128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:17.980128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:17.980140 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:17.982139 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:17.982521 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:17.982524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:17.982524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:17.982631 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:17.982837 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:17.982838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:17.982838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:17.983129 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:17.983295 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:17.983359 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 05:09:17.983498 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260925 05:09:17.997254 31483 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:17.997748 31483 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:18.002661 31483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36: Generated new TSK 0
I20260925 05:09:18.002843 31483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:09:18.027408 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36937
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:18.106977 31496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:18.107172 31496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:18.107194 31496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:18.108325 31496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:18.108364 31496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:18.108390 31496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:18.109800 31496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:18.110029 31496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:18.110287 31496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:18.115103 31502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.115119 31504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.115092 31501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:18.115307 31496 server_base.cc:1034] running on GCE node
I20260925 05:09:18.115530 31496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:18.115769 31496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:18.116931 31496 hybrid_clock.cc:648] HybridClock initialized: now 1790312958116917 us; error 29 us; skew 500 ppm
Sep 25 05:09:18 dist-test-slave-4dmq krb5kdc[31389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:18.119359 31496 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:09:18.119751 31496 webserver.cc:460] Webserver started at http://127.14.33.1:41127/ using document root <none> and password file <none>
I20260925 05:09:18.119962 31496 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:18.120028 31496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:18.120163 31496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:18.121199 31496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "7867c7d8261d46509e4077a6ac5b0384"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "92a762f8df071984c656de6f27efc366"
server_key_iv: "edf402d9544d36cf3a8513bb97dc7069"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.121567 31496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "7867c7d8261d46509e4077a6ac5b0384"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "92a762f8df071984c656de6f27efc366"
server_key_iv: "edf402d9544d36cf3a8513bb97dc7069"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.122779 31496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:18.123456 31511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.123728 31496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:18.123821 31496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "7867c7d8261d46509e4077a6ac5b0384"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "92a762f8df071984c656de6f27efc366"
server_key_iv: "edf402d9544d36cf3a8513bb97dc7069"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.123896 31496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:18.146888 31496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:18.149094 31496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:18.149259 31496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:18.149467 31496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:18.149742 31496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:18.149789 31496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.149820 31496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:18.149838 31496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.155377 31496 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:37845
I20260925 05:09:18.155417 31624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:37845 every 8 connection(s)
I20260925 05:09:18.155714 31496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
Sep 25 05:09:18 dist-test-slave-4dmq krb5kdc[31389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:18.162074 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31496
I20260925 05:09:18.162228 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:18.162436 14468 external_mini_cluster.cc:1428] Setting key b88d48d2f52d33aeec7cf4450dc5e94c
I20260925 05:09:18.164691 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.156395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49913 (local address 127.14.33.62:36937)
0925 05:09:18.156446 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.156449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.156644 (+ 195us) server_negotiation.cc:407] Connection header received
0925 05:09:18.157073 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.157074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.157100 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.157230 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:18.158076 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.158573 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.159488 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.159681 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.159727 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.161661 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:18.161662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:18.161663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:18.161669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:18.163223 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.163627 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.163628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.163629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.163690 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.163977 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.163978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.163978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.164087 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:18.164161 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.164548 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.164626 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 05:09:18.164922 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.156370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36937 (local address 127.14.33.1:49913)
0925 05:09:18.156532 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.156565 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.156993 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.157285 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.157288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.157830 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:18.158000 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.158004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.158647 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.158647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.159417 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.159419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.159745 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.159746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.159833 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.160322 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:18.160330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:18.161580 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:18.163321 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:18.163329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:18.163331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:18.163554 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:18.163754 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:18.163755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:18.163756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:18.163894 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:18.164190 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:18.164197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:18.164372 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.164515 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.164680 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 05:09:18.165444 31625 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36937
I20260925 05:09:18.165531 31625 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:18.165701 31625 heartbeater.cc:511] Master 127.14.33.62:36937 requested a full tablet report, sending...
I20260925 05:09:18.166270 31434 ts_manager.cc:194] Registered new tserver with Master: 7867c7d8261d46509e4077a6ac5b0384 (127.14.33.1:37845)
I20260925 05:09:18.167222 31434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:49913
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:09:18.188246 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36937
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:18.271960 31630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:18.272141 31630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:18.272163 31630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:18.273387 31630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:18.273427 31630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:18.273469 31630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:18.274855 31630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:18.275079 31630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:18.275295 31630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:18.279985 31638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.280047 31636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.280014 31635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:18.280181 31630 server_base.cc:1034] running on GCE node
I20260925 05:09:18.280395 31630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:18.280666 31630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:18.281805 31630 hybrid_clock.cc:648] HybridClock initialized: now 1790312958281791 us; error 25 us; skew 500 ppm
Sep 25 05:09:18 dist-test-slave-4dmq krb5kdc[31389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:18.284019 31630 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:09:18.284430 31630 webserver.cc:460] Webserver started at http://127.14.33.2:43377/ using document root <none> and password file <none>
I20260925 05:09:18.284657 31630 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:18.284719 31630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:18.284860 31630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:18.285950 31630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "ae4cb9fd7bce4454a523e12ec30af746"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "4d9ad2f63378d78bf6de4f40aa2ed49c"
server_key_iv: "e133b096130b470a722ea662b6dee39f"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.286320 31630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "ae4cb9fd7bce4454a523e12ec30af746"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "4d9ad2f63378d78bf6de4f40aa2ed49c"
server_key_iv: "e133b096130b470a722ea662b6dee39f"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.287616 31630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:09:18.288295 31645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.288471 31630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:18.288563 31630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "ae4cb9fd7bce4454a523e12ec30af746"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "4d9ad2f63378d78bf6de4f40aa2ed49c"
server_key_iv: "e133b096130b470a722ea662b6dee39f"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.288642 31630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:18.299166 31630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:18.300984 31630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:18.301206 31630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:18.301435 31630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:18.301728 31630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:18.301777 31630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.301818 31630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:18.301846 31630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.307488 31630 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:34173
I20260925 05:09:18.307534 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:34173 every 8 connection(s)
I20260925 05:09:18.307816 31630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:18.313160 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31630
I20260925 05:09:18.313279 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:18.313452 14468 external_mini_cluster.cc:1428] Setting key 67b0f8dc1952fda1dcf4656a8004feb6
Sep 25 05:09:18 dist-test-slave-4dmq krb5kdc[31389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:18.317257 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.308473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:54163 (local address 127.14.33.62:36937)
0925 05:09:18.308537 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.308540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.308723 (+ 183us) server_negotiation.cc:407] Connection header received
0925 05:09:18.309250 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.309251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.309265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.309313 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:18.310240 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.310740 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.311747 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.311955 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.312050 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.314293 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:18.314295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:18.314296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:18.314304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:18.315931 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.316299 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.316301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.316301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.316361 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.316628 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.316629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.316629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.316728 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:18.316779 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.317081 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.317189 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:09:18.317546 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.308451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36937 (local address 127.14.33.2:54163)
0925 05:09:18.308605 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.308641 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.309146 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.309400 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.309403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.309951 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:18.310137 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.310141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.310864 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.310865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.311652 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.311655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.312058 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.312059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.312180 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.312674 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:18.312682 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:18.314202 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:18.316016 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:18.316022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:18.316023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:18.316228 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:18.316428 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:18.316429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:18.316429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:18.316565 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:18.316788 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:18.316794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:18.316952 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.317064 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.317264 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260925 05:09:18.318038 31759 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36937
I20260925 05:09:18.318148 31759 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:18.318321 31759 heartbeater.cc:511] Master 127.14.33.62:36937 requested a full tablet report, sending...
I20260925 05:09:18.318691 31434 ts_manager.cc:194] Registered new tserver with Master: ae4cb9fd7bce4454a523e12ec30af746 (127.14.33.2:34173)
I20260925 05:09:18.319061 31434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:54163
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:09:18.338008 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36937
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:18.416848 31764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:18.417083 31764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:18.417119 31764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:18.418215 31764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:18.418282 31764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:18.418323 31764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:18.419739 31764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:36937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:18.420008 31764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:18.420228 31764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:18.424842 31770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.424868 31769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:18.424971 31764 server_base.cc:1034] running on GCE node
W20260925 05:09:18.424844 31772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:18.425300 31764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:18.425537 31764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:18.426694 31764 hybrid_clock.cc:648] HybridClock initialized: now 1790312958426684 us; error 25 us; skew 500 ppm
Sep 25 05:09:18 dist-test-slave-4dmq krb5kdc[31389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:18.429031 31764 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:09:18.429451 31764 webserver.cc:460] Webserver started at http://127.14.33.3:33463/ using document root <none> and password file <none>
I20260925 05:09:18.429669 31764 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:18.429730 31764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:18.429870 31764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:18.430792 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "a2bf7de124d1452b8e16c87c480e10e1"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "d66187ea9ab038ee88276d4c44c4e2da"
server_key_iv: "54faabcea388913de6a90a4fdb25b99e"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.431121 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "a2bf7de124d1452b8e16c87c480e10e1"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "d66187ea9ab038ee88276d4c44c4e2da"
server_key_iv: "54faabcea388913de6a90a4fdb25b99e"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.432302 31764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:18.432929 31779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.433084 31764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:18.433192 31764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "a2bf7de124d1452b8e16c87c480e10e1"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "d66187ea9ab038ee88276d4c44c4e2da"
server_key_iv: "54faabcea388913de6a90a4fdb25b99e"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.433267 31764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:18.454216 31764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:18.456683 31764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:18.456845 31764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:18.457093 31764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:18.457486 31764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:18.457523 31764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.457547 31764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:18.457561 31764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.463768 31764 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:32771
I20260925 05:09:18.463796 31892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:32771 every 8 connection(s)
I20260925 05:09:18.464129 31764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:09:18 dist-test-slave-4dmq krb5kdc[31389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:18.472103 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31764
I20260925 05:09:18.472218 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:18.472445 14468 external_mini_cluster.cc:1428] Setting key fc4badc0b09a12c4a20d47666eeec8f0
I20260925 05:09:18.473621 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.465042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:56843 (local address 127.14.33.62:36937)
0925 05:09:18.465099 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.465101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.465132 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:09:18.465675 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.465677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.465685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.465734 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:18.466659 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.467184 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.468095 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.468264 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.468311 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.470460 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:18.470462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:18.470462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:18.470468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:18.471901 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.472494 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.472495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.472496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.472560 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.472894 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.472895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.472896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.473045 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:18.473128 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.473452 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.473544 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:09:18.473773 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.464769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:36937 (local address 127.14.33.3:56843)
0925 05:09:18.465001 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.465059 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.465590 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.465803 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.465807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.466362 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:18.466592 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.466597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.467260 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.467261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.468027 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.468029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.468333 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.468334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.468428 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.468956 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:18.468964 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:18.470381 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:18.471996 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:18.472001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:18.472003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:18.472214 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:18.472632 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:18.472633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:18.472634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:18.472780 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:18.473129 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:18.473156 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:18.473310 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.473424 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.473590 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260925 05:09:18.474171 31893 heartbeater.cc:347] Connected to a master server at 127.14.33.62:36937
I20260925 05:09:18.474257 31893 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:18.474404 31893 heartbeater.cc:511] Master 127.14.33.62:36937 requested a full tablet report, sending...
I20260925 05:09:18.474705 31434 ts_manager.cc:194] Registered new tserver with Master: a2bf7de124d1452b8e16c87c480e10e1 (127.14.33.3:32771)
I20260925 05:09:18.475024 31434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:56843
I20260925 05:09:18.484263 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:18.490904 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.485498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.14.33.62:36937)
0925 05:09:18.485567 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.485569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.485671 (+ 102us) server_negotiation.cc:407] Connection header received
0925 05:09:18.485752 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.485754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.485763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.485817 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:18.486327 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.486786 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.487479 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.487673 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.487746 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.488772 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:18.488773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:18.488774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:18.488780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:18.489859 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.490163 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.490164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.490165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.490215 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:18.490394 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:18.490394 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:18.490395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:18.490519 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:18.490612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.490726 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.490801 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:09:18.492342 31434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:18.493880 31434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:18.499969 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.495903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:34173 (local address 127.0.0.1:48894)
0925 05:09:18.496229 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.496239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.496457 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.496731 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.496731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.496733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:18.496771 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.496773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.498025 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.498026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.498950 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.498952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.499649 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.499728 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.499746 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.499846 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.499899 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":58,"threads_started":1}
I20260925 05:09:18.499986 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.496009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.14.33.2:34173)
0925 05:09:18.496182 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.496187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.496294 (+ 107us) server_negotiation.cc:407] Connection header received
0925 05:09:18.496555 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.496557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.496571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.496636 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:18.496864 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.497922 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499027 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.499557 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499640 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.499699 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.499864 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.499905 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260925 05:09:18.500139 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.496091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:37845 (local address 127.0.0.1:43362)
0925 05:09:18.496348 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.496358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.496401 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.496712 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.496713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.496715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:18.496771 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.496773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.497664 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.497665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.498853 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.498856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.499815 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.499875 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.499879 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.499925 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.499967 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":51,"threads_started":1}
I20260925 05:09:18.500161 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.496290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.14.33.1:37845)
0925 05:09:18.496491 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.496495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.496505 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:09:18.496544 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.496546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.496560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.496646 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:18.496861 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.497571 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.498941 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.499659 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499727 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.499779 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.500017 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.500063 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260925 05:09:18.500285 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.495963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.14.33.3:32771)
0925 05:09:18.496105 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.496110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.496272 (+ 162us) server_negotiation.cc:407] Connection header received
0925 05:09:18.496370 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.496373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.496395 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.496488 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:18.496810 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.497903 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499160 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.499800 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499857 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.499917 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.500168 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.500211 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260925 05:09:18.500487 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.495890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:32771 (local address 127.0.0.1:40170)
0925 05:09:18.496180 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.496208 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.496291 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.496638 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.496641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.496643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:18.496702 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.496705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.498011 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.498012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.499088 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.499090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.499946 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.499947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.499998 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.500001 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.500385 (+ 384us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.500429 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":22,"thread_start_us":57,"threads_started":1}
I20260925 05:09:18.500876 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 1d74339d8c9b45e69215860e0eab74ab (DEFAULT_TABLE table=test-table [id=b1cb9688b46c4c8f997b632a82ad8f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:18.501088 31827 tablet_service.cc:1467] Processing CreateTablet for tablet 1d74339d8c9b45e69215860e0eab74ab (DEFAULT_TABLE table=test-table [id=b1cb9688b46c4c8f997b632a82ad8f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:18.501082 31693 tablet_service.cc:1467] Processing CreateTablet for tablet 1d74339d8c9b45e69215860e0eab74ab (DEFAULT_TABLE table=test-table [id=b1cb9688b46c4c8f997b632a82ad8f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:18.501257 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d74339d8c9b45e69215860e0eab74ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:18.501410 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d74339d8c9b45e69215860e0eab74ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:18.501423 31827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d74339d8c9b45e69215860e0eab74ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:18.504441 31909 tablet_bootstrap.cc:492] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Bootstrap starting.
I20260925 05:09:18.504532 31907 tablet_bootstrap.cc:492] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Bootstrap starting.
I20260925 05:09:18.505028 31909 tablet_bootstrap.cc:654] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:18.505082 31907 tablet_bootstrap.cc:654] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:18.505535 31909 log.cc:824] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:18.505563 31907 log.cc:824] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:18.505872 31908 tablet_bootstrap.cc:492] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: Bootstrap starting.
I20260925 05:09:18.506306 31909 tablet_bootstrap.cc:492] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: No bootstrap required, opened a new log
I20260925 05:09:18.506290 31907 tablet_bootstrap.cc:492] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: No bootstrap required, opened a new log
I20260925 05:09:18.506372 31909 ts_tablet_manager.cc:1397] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:09:18.506372 31907 ts_tablet_manager.cc:1397] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:09:18.506446 31908 tablet_bootstrap.cc:654] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:18.506959 31908 log.cc:824] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:18.507588 31908 tablet_bootstrap.cc:492] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: No bootstrap required, opened a new log
I20260925 05:09:18.507663 31908 ts_tablet_manager.cc:1397] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:18.508241 31907 raft_consensus.cc:348] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.508241 31909 raft_consensus.cc:348] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.508419 31907 raft_consensus.cc:374] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:18.508419 31909 raft_consensus.cc:374] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:18.508455 31909 raft_consensus.cc:729] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4cb9fd7bce4454a523e12ec30af746, State: Initialized, Role: FOLLOWER
I20260925 05:09:18.508455 31907 raft_consensus.cc:729] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bf7de124d1452b8e16c87c480e10e1, State: Initialized, Role: FOLLOWER
I20260925 05:09:18.508587 31907 consensus_queue.cc:260] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.508586 31909 consensus_queue.cc:260] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.508843 31909 ts_tablet_manager.cc:1428] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:09:18.508837 31907 ts_tablet_manager.cc:1428] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 05:09:18.508947 31893 heartbeater.cc:503] Master 127.14.33.62:36937 was elected leader, sending a full tablet report...
I20260925 05:09:18.509001 31759 heartbeater.cc:503] Master 127.14.33.62:36937 was elected leader, sending a full tablet report...
I20260925 05:09:18.509271 31908 raft_consensus.cc:348] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.509438 31908 raft_consensus.cc:374] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:18.509483 31908 raft_consensus.cc:729] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7867c7d8261d46509e4077a6ac5b0384, State: Initialized, Role: FOLLOWER
I20260925 05:09:18.509593 31908 consensus_queue.cc:260] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.509855 31625 heartbeater.cc:503] Master 127.14.33.62:36937 was elected leader, sending a full tablet report...
I20260925 05:09:18.509840 31908 ts_tablet_manager.cc:1428] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 05:09:18.558301 31760 tablet.cc:2369] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:18.581071 31914 raft_consensus.cc:482] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:18.581288 31914 raft_consensus.cc:504] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.581670 31914 leader_election.cc:288] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4cb9fd7bce4454a523e12ec30af746 (127.14.33.2:34173), 7867c7d8261d46509e4077a6ac5b0384 (127.14.33.1:37845)
I20260925 05:09:18.584714 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.581943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:34145 (local address 127.14.33.1:37845)
0925 05:09:18.581985 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.581987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.582304 (+ 317us) server_negotiation.cc:407] Connection header received
0925 05:09:18.582364 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.582366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.582378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.582433 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:18.582570 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.583010 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.583927 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.584386 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.584454 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.584465 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.584612 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.584647 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:18.584758 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.581980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:52389 (local address 127.14.33.2:34173)
0925 05:09:18.582183 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.582186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.582197 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:09:18.582229 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.582231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.582243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.582323 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:18.582497 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.582971 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.583968 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.584392 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.584467 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.584477 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.584644 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.584681 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260925 05:09:18.584748 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.582067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:37845 (local address 127.14.33.3:34145)
0925 05:09:18.582241 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.582249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.582312 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.582435 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.582435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.582436 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:18.582502 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.582504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.583085 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.583086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.583833 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.583834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.584465 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.584466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.584540 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.584543 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.584595 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.584636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":129,"threads_started":1}
I20260925 05:09:18.584759 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.581818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:34173 (local address 127.14.33.3:52389)
0925 05:09:18.581876 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.581901 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.581946 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.582335 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.582337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.582339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:18.582415 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.582417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.583055 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.583056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.583836 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.583839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.584505 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.584506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.584566 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.584569 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.584638 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.584677 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":35}
I20260925 05:09:18.585233 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d74339d8c9b45e69215860e0eab74ab" candidate_uuid: "a2bf7de124d1452b8e16c87c480e10e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4cb9fd7bce4454a523e12ec30af746" is_pre_election: true
I20260925 05:09:18.585378 31713 raft_consensus.cc:2454] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bf7de124d1452b8e16c87c480e10e1 in term 0.
I20260925 05:09:18.585367 31579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d74339d8c9b45e69215860e0eab74ab" candidate_uuid: "a2bf7de124d1452b8e16c87c480e10e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7867c7d8261d46509e4077a6ac5b0384" is_pre_election: true
I20260925 05:09:18.585526 31579 raft_consensus.cc:2454] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2bf7de124d1452b8e16c87c480e10e1 in term 0.
I20260925 05:09:18.585605 31780 leader_election.cc:302] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2bf7de124d1452b8e16c87c480e10e1, ae4cb9fd7bce4454a523e12ec30af746; no voters:
I20260925 05:09:18.585748 31914 raft_consensus.cc:2781] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:18.585804 31914 raft_consensus.cc:482] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:18.585840 31914 raft_consensus.cc:3037] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:18.586747 31914 raft_consensus.cc:504] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.586897 31914 leader_election.cc:288] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [CANDIDATE]: Term 1 election: Requested vote from peers ae4cb9fd7bce4454a523e12ec30af746 (127.14.33.2:34173), 7867c7d8261d46509e4077a6ac5b0384 (127.14.33.1:37845)
I20260925 05:09:18.587044 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d74339d8c9b45e69215860e0eab74ab" candidate_uuid: "a2bf7de124d1452b8e16c87c480e10e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4cb9fd7bce4454a523e12ec30af746"
I20260925 05:09:18.587047 31579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d74339d8c9b45e69215860e0eab74ab" candidate_uuid: "a2bf7de124d1452b8e16c87c480e10e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7867c7d8261d46509e4077a6ac5b0384"
I20260925 05:09:18.587116 31713 raft_consensus.cc:3037] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:18.587123 31579 raft_consensus.cc:3037] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:18.587826 31713 raft_consensus.cc:2454] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2bf7de124d1452b8e16c87c480e10e1 in term 1.
I20260925 05:09:18.587883 31579 raft_consensus.cc:2454] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2bf7de124d1452b8e16c87c480e10e1 in term 1.
I20260925 05:09:18.588027 31780 leader_election.cc:302] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2bf7de124d1452b8e16c87c480e10e1, ae4cb9fd7bce4454a523e12ec30af746; no voters:
I20260925 05:09:18.588129 31914 raft_consensus.cc:2781] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:18.588290 31914 raft_consensus.cc:686] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 1 LEADER]: Becoming Leader. State: Replica: a2bf7de124d1452b8e16c87c480e10e1, State: Running, Role: LEADER
I20260925 05:09:18.588411 31914 consensus_queue.cc:237] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } }
I20260925 05:09:18.589236 31434 catalog_manager.cc:5495] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2bf7de124d1452b8e16c87c480e10e1 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "a2bf7de124d1452b8e16c87c480e10e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bf7de124d1452b8e16c87c480e10e1" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 32771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 } health_report { overall_health: UNKNOWN } } }
I20260925 05:09:18.613479 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.611242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.14.33.3:32771)
0925 05:09:18.611284 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.611286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.611294 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:18.611325 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.611326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.611344 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.611421 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:18.611532 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.611977 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.612708 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.612908 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.612976 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.613148 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:18.613237 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:18.613324 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.613361 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.613380 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:09:18.615113 31579 raft_consensus.cc:1261] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 1 FOLLOWER]: Refusing update from remote peer a2bf7de124d1452b8e16c87c480e10e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:18.615113 31713 raft_consensus.cc:1261] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 1 FOLLOWER]: Refusing update from remote peer a2bf7de124d1452b8e16c87c480e10e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:18.615454 31914 consensus_queue.cc:1035] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4cb9fd7bce4454a523e12ec30af746" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:18.615613 31917 consensus_queue.cc:1035] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7867c7d8261d46509e4077a6ac5b0384" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:18.618331 31921 mvcc.cc:205] Tried to move back new op lower bound from 7333121878484955136 to 7333121878378872832. Current Snapshot: MvccSnapshot[applied={T|T < 7333121878484955136}]
I20260925 05:09:18.620126 31432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:18.620209 31432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:18.620947 31432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c87c9be4e3d4da2a1193dfbfe8e3d36: Sending DeleteTablet for 3 replicas of tablet 1d74339d8c9b45e69215860e0eab74ab
I20260925 05:09:18.621275 31693 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d74339d8c9b45e69215860e0eab74ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:18 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:48894
I20260925 05:09:18.621312 31559 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d74339d8c9b45e69215860e0eab74ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:18 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:43362
I20260925 05:09:18.621315 31827 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d74339d8c9b45e69215860e0eab74ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:18 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:40170
I20260925 05:09:18.621532 31931 tablet_replica.cc:330] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384: stopping tablet replica
I20260925 05:09:18.621634 31931 raft_consensus.cc:2229] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:18.621719 31931 raft_consensus.cc:2258] T 1d74339d8c9b45e69215860e0eab74ab P 7867c7d8261d46509e4077a6ac5b0384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:18.621811 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31496
I20260925 05:09:18.622797 31930 tablet_replica.cc:330] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: stopping tablet replica
I20260925 05:09:18.622901 31930 raft_consensus.cc:2229] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:18.622990 31930 raft_consensus.cc:2258] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:18.623129 31932 tablet_replica.cc:330] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: stopping tablet replica
I20260925 05:09:18.623209 31932 raft_consensus.cc:2229] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:09:18.623325 31932 raft_consensus.cc:2258] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:18.623544 31930 ts_tablet_manager.cc:1905] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:18.623772 31932 ts_tablet_manager.cc:1905] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:18.625545 31932 ts_tablet_manager.cc:1918] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:18.625557 31930 ts_tablet_manager.cc:1918] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:18.625633 31932 log.cc:1196] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/1d74339d8c9b45e69215860e0eab74ab
I20260925 05:09:18.625741 31930 log.cc:1196] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/1d74339d8c9b45e69215860e0eab74ab
I20260925 05:09:18.625944 31932 ts_tablet_manager.cc:1939] T 1d74339d8c9b45e69215860e0eab74ab P a2bf7de124d1452b8e16c87c480e10e1: Deleting consensus metadata
I20260925 05:09:18.626170 31930 ts_tablet_manager.cc:1939] T 1d74339d8c9b45e69215860e0eab74ab P ae4cb9fd7bce4454a523e12ec30af746: Deleting consensus metadata
I20260925 05:09:18.626359 31420 catalog_manager.cc:4841] TS a2bf7de124d1452b8e16c87c480e10e1 (127.14.33.3:32771): tablet 1d74339d8c9b45e69215860e0eab74ab (table test-table [id=b1cb9688b46c4c8f997b632a82ad8f0f]) successfully deleted
I20260925 05:09:18.626672 31418 catalog_manager.cc:4841] TS ae4cb9fd7bce4454a523e12ec30af746 (127.14.33.2:34173): tablet 1d74339d8c9b45e69215860e0eab74ab (table test-table [id=b1cb9688b46c4c8f997b632a82ad8f0f]) successfully deleted
W20260925 05:09:18.627566 31420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:37845 (error 108)
I20260925 05:09:18.627745 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31630
I20260925 05:09:18.632537 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.632385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:37845 (local address 127.0.0.1:43376)
0925 05:09:18.632445 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.632486 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:37845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 05:09:18.632541 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31764
W20260925 05:09:18.632664 31420 catalog_manager.cc:4584] TS 7867c7d8261d46509e4077a6ac5b0384 (127.14.33.1:37845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d74339d8c9b45e69215860e0eab74ab: Network error: Client connection negotiation failed: client connection to 127.14.33.1:37845: connect: Connection refused (error 111)
I20260925 05:09:18.637511 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31402
2026-09-25T05:09:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1914 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T05:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:18Z Disabled control of system clock
I20260925 05:09:18.664036 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41985
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:18.739519 31937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:18.739727 31937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:18.739746 31937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:18.740895 31937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:18.740935 31937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:18.740948 31937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:18.740995 31937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:18.741019 31937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:18.742329 31937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:18.742568 31937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:18.742854 31937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:18.747498 31945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.747510 31942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.747510 31943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:18.747741 31937 server_base.cc:1034] running on GCE node
I20260925 05:09:18.747992 31937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:18.748229 31937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:18.749362 31937 hybrid_clock.cc:648] HybridClock initialized: now 1790312958749350 us; error 27 us; skew 500 ppm
I20260925 05:09:18.750496 31937 webserver.cc:460] Webserver started at http://127.14.33.62:35275/ using document root <none> and password file <none>
I20260925 05:09:18.750726 31937 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:18.750770 31937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:18.750927 31937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:18.751902 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "13a6b619e06946e289096cc154cfa07b"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "83abf4eb00a4db53c6c506af438b6df5"
server_key_iv: "b80bd5a652b2395de046f0de6bc00d9d"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.752228 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "13a6b619e06946e289096cc154cfa07b"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "83abf4eb00a4db53c6c506af438b6df5"
server_key_iv: "b80bd5a652b2395de046f0de6bc00d9d"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.753453 31937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:18.754099 31951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.754258 31937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:18.754323 31937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "13a6b619e06946e289096cc154cfa07b"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "83abf4eb00a4db53c6c506af438b6df5"
server_key_iv: "b80bd5a652b2395de046f0de6bc00d9d"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.754410 31937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:18.770919 31937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:18.771206 31937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:18.771349 31937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:18.774959 31937 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:41985
I20260925 05:09:18.775003 32003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:41985 every 8 connection(s)
I20260925 05:09:18.775292 31937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:18.776213 32004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:18.777896 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 31937
I20260925 05:09:18.778003 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:18.778189 14468 external_mini_cluster.cc:1428] Setting key a981dec12a8ef179ecef2c8569a147df
I20260925 05:09:18.778867 32004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: Bootstrap starting.
I20260925 05:09:18.779567 32004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:18.779897 32004 log.cc:824] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:18.780607 32004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: No bootstrap required, opened a new log
I20260925 05:09:18.782693 32004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a6b619e06946e289096cc154cfa07b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41985 } }
I20260925 05:09:18.782851 32004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:18.783185 32004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a6b619e06946e289096cc154cfa07b, State: Initialized, Role: FOLLOWER
I20260925 05:09:18.783339 32004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a6b619e06946e289096cc154cfa07b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41985 } }
I20260925 05:09:18.783437 32004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:18.783480 32004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:18.783536 32004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:18.784020 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.778723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.14.33.62:41985)
0925 05:09:18.778928 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.778938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.778970 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:09:18.779563 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.779572 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.779651 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.779822 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:18.779962 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.781226 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.781943 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.782586 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.782716 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.783318 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:18.783322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:18.783322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:18.783523 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:18.783564 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:18.783577 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:18.783640 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.783735 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.783815 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260925 05:09:18.784174 32004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a6b619e06946e289096cc154cfa07b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41985 } }
I20260925 05:09:18.784278 32004 leader_election.cc:302] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a6b619e06946e289096cc154cfa07b; no voters:
I20260925 05:09:18.784440 32004 leader_election.cc:288] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:18.784469 32009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:18.784617 32009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [term 1 LEADER]: Becoming Leader. State: Replica: 13a6b619e06946e289096cc154cfa07b, State: Running, Role: LEADER
I20260925 05:09:18.784674 32004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:18.784726 32009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a6b619e06946e289096cc154cfa07b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41985 } }
I20260925 05:09:18.785096 32010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13a6b619e06946e289096cc154cfa07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a6b619e06946e289096cc154cfa07b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41985 } } }
I20260925 05:09:18.785209 32010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:18.785377 32011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13a6b619e06946e289096cc154cfa07b. Latest consensus state: current_term: 1 leader_uuid: "13a6b619e06946e289096cc154cfa07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a6b619e06946e289096cc154cfa07b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41985 } } }
I20260925 05:09:18.785447 32011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:18.786303 32025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:18.786360 32025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:18.786404 32018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:18.786556 32018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:18.787828 32018 catalog_manager.cc:1347] Generated new cluster ID: 9900770d34204f9a89107687279daa8d
I20260925 05:09:18.787882 32018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:18.816550 32018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:18.817292 32018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:18.823649 32018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: Generated new TSK 0
I20260925 05:09:18.823822 32018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:18.824780 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41985
--builtin_ntp_servers=127.14.33.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:18.902292 32028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:18.902487 32028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:18.902509 32028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:18.903620 32028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:18.903658 32028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:18.903681 32028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:18.904978 32028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:18.905256 32028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:18.905550 32028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:18.910329 32033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.910315 32036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:18.910328 32034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:18.910612 32028 server_base.cc:1034] running on GCE node
I20260925 05:09:18.910816 32028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:18.911036 32028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:18.912207 32028 hybrid_clock.cc:648] HybridClock initialized: now 1790312958912196 us; error 26 us; skew 500 ppm
I20260925 05:09:18.913640 32028 webserver.cc:460] Webserver started at http://127.14.33.1:36103/ using document root <none> and password file <none>
I20260925 05:09:18.913866 32028 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:18.913911 32028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:18.914055 32028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:18.915047 32028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "fd53222e5c8247a5b182c85bb9a51732"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "4bf95ff597c09ff1b5d5106666be3023"
server_key_iv: "9f1e6737fb7212268355d70797cbb4e9"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.915385 32028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "fd53222e5c8247a5b182c85bb9a51732"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "4bf95ff597c09ff1b5d5106666be3023"
server_key_iv: "9f1e6737fb7212268355d70797cbb4e9"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.916621 32028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:18.917339 32042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.917493 32028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:18.917580 32028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "fd53222e5c8247a5b182c85bb9a51732"
format_stamp: "Formatted at 2026-09-25 05:09:18 on dist-test-slave-4dmq"
server_key: "4bf95ff597c09ff1b5d5106666be3023"
server_key_iv: "9f1e6737fb7212268355d70797cbb4e9"
server_key_version: "encryptionkey@0"
I20260925 05:09:18.917656 32028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:18.928375 32028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:18.928635 32028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:18.928769 32028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:18.928998 32028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:18.929348 32028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:18.929400 32028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.929440 32028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:18.929466 32028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:18.935410 32028 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:39187
I20260925 05:09:18.935446 32155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:39187 every 8 connection(s)
I20260925 05:09:18.935765 32028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:18.939013 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 32028
I20260925 05:09:18.939095 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:18.939312 14468 external_mini_cluster.cc:1428] Setting key 61d375dfbdeab5db9fff3a4c4c941a09
I20260925 05:09:18.940222 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41985
--builtin_ntp_servers=127.14.33.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:09:18.940644 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.936334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:37259 (local address 127.14.33.62:41985)
0925 05:09:18.936392 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:18.936394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:18.936603 (+ 209us) server_negotiation.cc:407] Connection header received
0925 05:09:18.937052 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:18.937053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:18.937067 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:18.937150 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:18.937768 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.938256 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.939199 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.939494 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.939604 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.940401 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:18.940402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:18.940403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:18.940407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:18.940421 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:18.940426 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:18.940451 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:18.940565 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:09:18.940587 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:09:18.940937 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:18.936325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41985 (local address 127.14.33.1:37259)
0925 05:09:18.936490 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:18.936521 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:18.936974 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:18.937234 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:18.937237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:18.937245 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:18.937701 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.937705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.938325 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.938326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.939054 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:18.939075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:18.939614 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:18.939615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:18.939771 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:18.940235 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:18.940236 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:18.940273 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:18.940300 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:18.940487 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:18.940492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:18.940494 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:18.940554 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 05:09:18.940660 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260925 05:09:18.941874 32156 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41985
I20260925 05:09:18.941969 32156 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:18.942134 32156 heartbeater.cc:511] Master 127.14.33.62:41985 requested a full tablet report, sending...
I20260925 05:09:18.942629 31968 ts_manager.cc:194] Registered new tserver with Master: fd53222e5c8247a5b182c85bb9a51732 (127.14.33.1:39187)
I20260925 05:09:18.943207 31968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.1:37259
W20260925 05:09:19.016774 32159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:19.016968 32159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:19.017000 32159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:19.018066 32159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:19.018121 32159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:19.018160 32159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:19.019408 32159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:19.019631 32159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:19.019917 32159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:19.024822 32165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:19.024847 32167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:19.024919 32159 server_base.cc:1034] running on GCE node
W20260925 05:09:19.024783 32164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:19.025210 32159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:19.025453 32159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:19.026587 32159 hybrid_clock.cc:648] HybridClock initialized: now 1790312959026569 us; error 34 us; skew 500 ppm
I20260925 05:09:19.027757 32159 webserver.cc:460] Webserver started at http://127.14.33.2:40903/ using document root <none> and password file <none>
I20260925 05:09:19.027973 32159 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:19.028017 32159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:19.028174 32159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:19.029181 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "4656836959ff4de0bac9604814cbb8d5"
format_stamp: "Formatted at 2026-09-25 05:09:19 on dist-test-slave-4dmq"
server_key: "46976ca153837c1cfed3ec29429d32cf"
server_key_iv: "7d1228cedb13a883aa371d906cdfad6a"
server_key_version: "encryptionkey@0"
I20260925 05:09:19.029531 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "4656836959ff4de0bac9604814cbb8d5"
format_stamp: "Formatted at 2026-09-25 05:09:19 on dist-test-slave-4dmq"
server_key: "46976ca153837c1cfed3ec29429d32cf"
server_key_iv: "7d1228cedb13a883aa371d906cdfad6a"
server_key_version: "encryptionkey@0"
I20260925 05:09:19.030712 32159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:19.031325 32173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:19.031469 32159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:19.031544 32159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "4656836959ff4de0bac9604814cbb8d5"
format_stamp: "Formatted at 2026-09-25 05:09:19 on dist-test-slave-4dmq"
server_key: "46976ca153837c1cfed3ec29429d32cf"
server_key_iv: "7d1228cedb13a883aa371d906cdfad6a"
server_key_version: "encryptionkey@0"
I20260925 05:09:19.031606 32159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:19.048322 32159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:19.048593 32159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:19.048728 32159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:19.048921 32159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:19.049252 32159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:19.049286 32159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:19.049324 32159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:19.049348 32159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:19.054673 32159 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:44299
I20260925 05:09:19.054708 32286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:44299 every 8 connection(s)
I20260925 05:09:19.055009 32159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:19.059774 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.055642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:56305 (local address 127.14.33.62:41985)
0925 05:09:19.055683 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.055685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.055894 (+ 209us) server_negotiation.cc:407] Connection header received
0925 05:09:19.056483 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.056485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.056497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.056567 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:19.057244 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.057779 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.058694 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.058867 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.058917 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.059575 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:19.059576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:19.059576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:19.059580 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:19.059593 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:19.059597 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:19.059641 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.059699 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.059718 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:19.059953 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.055624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41985 (local address 127.14.33.2:56305)
0925 05:09:19.055779 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.055809 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.056373 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.056631 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.056635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.056643 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:19.057166 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.057172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.057888 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.057889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.058625 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.058627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.058933 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.058934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.059034 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.059461 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:19.059461 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:19.059487 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:19.059507 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:19.059644 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:19.059646 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:19.059648 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.059691 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.059761 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260925 05:09:19.060432 32287 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41985
I20260925 05:09:19.060524 32287 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:19.060685 32287 heartbeater.cc:511] Master 127.14.33.62:41985 requested a full tablet report, sending...
I20260925 05:09:19.061065 31968 ts_manager.cc:194] Registered new tserver with Master: 4656836959ff4de0bac9604814cbb8d5 (127.14.33.2:44299)
I20260925 05:09:19.061439 31968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.2:56305
I20260925 05:09:19.065243 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 32159
I20260925 05:09:19.065353 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:19.065537 14468 external_mini_cluster.cc:1428] Setting key 6cbd468b79a95636d4f9c60368b718e5
I20260925 05:09:19.066423 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41985
--builtin_ntp_servers=127.14.33.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:19.141371 32290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:19.141541 32290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:19.141562 32290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:19.142618 32290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:19.142656 32290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:19.142700 32290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:19.144014 32290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:19.144261 32290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:19.144549 32290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:19.149340 32295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:19.149350 32298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:19.149382 32296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:19.149387 32290 server_base.cc:1034] running on GCE node
I20260925 05:09:19.149761 32290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:19.149947 32290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:19.151096 32290 hybrid_clock.cc:648] HybridClock initialized: now 1790312959151084 us; error 32 us; skew 500 ppm
I20260925 05:09:19.152307 32290 webserver.cc:460] Webserver started at http://127.14.33.3:42733/ using document root <none> and password file <none>
I20260925 05:09:19.152552 32290 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:19.152621 32290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:19.152763 32290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:19.153846 32290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "6552d72f377c46f0ad938ac672612a13"
format_stamp: "Formatted at 2026-09-25 05:09:19 on dist-test-slave-4dmq"
server_key: "c17b6deb2c2128bc8d4f57967876e1ee"
server_key_iv: "1160052c44f1f64237da5daac9019380"
server_key_version: "encryptionkey@0"
I20260925 05:09:19.154176 32290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "6552d72f377c46f0ad938ac672612a13"
format_stamp: "Formatted at 2026-09-25 05:09:19 on dist-test-slave-4dmq"
server_key: "c17b6deb2c2128bc8d4f57967876e1ee"
server_key_iv: "1160052c44f1f64237da5daac9019380"
server_key_version: "encryptionkey@0"
I20260925 05:09:19.155364 32290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:19.156044 32304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:19.156195 32290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:19.156286 32290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "6552d72f377c46f0ad938ac672612a13"
format_stamp: "Formatted at 2026-09-25 05:09:19 on dist-test-slave-4dmq"
server_key: "c17b6deb2c2128bc8d4f57967876e1ee"
server_key_iv: "1160052c44f1f64237da5daac9019380"
server_key_version: "encryptionkey@0"
I20260925 05:09:19.156395 32290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:19.173698 32290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:19.173954 32290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:19.174077 32290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:19.174281 32290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:19.174571 32290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:19.174619 32290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:19.174650 32290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:19.174677 32290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:19.180111 32290 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:37573
I20260925 05:09:19.180151 32417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:37573 every 8 connection(s)
I20260925 05:09:19.180441 32290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:09:19.180699 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 32290
I20260925 05:09:19.180817 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:19.181008 14468 external_mini_cluster.cc:1428] Setting key eb5147c1060b0296a7657dbc525ccbc4
I20260925 05:09:19.185863 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.181092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:53699 (local address 127.14.33.62:41985)
0925 05:09:19.181163 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.181166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.181387 (+ 221us) server_negotiation.cc:407] Connection header received
0925 05:09:19.182024 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.182025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.182037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.182090 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:19.182711 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.183207 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.184386 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.184687 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.184772 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.185613 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:19.185614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:19.185615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:19.185620 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:19.185636 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:19.185639 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:19.185691 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.185758 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.185784 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:09:19.186059 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.181078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41985 (local address 127.14.33.3:53699)
0925 05:09:19.181278 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.181319 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.181939 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.182156 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.182159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.182168 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:19.182624 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.182628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.183424 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.183425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.184289 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.184292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.184777 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.184778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.184977 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.185461 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:19.185461 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:19.185487 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:19.185507 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:19.185694 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:19.185699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:19.185702 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.185763 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.185866 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260925 05:09:19.186522 32418 heartbeater.cc:347] Connected to a master server at 127.14.33.62:41985
I20260925 05:09:19.186622 32418 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:19.186789 32418 heartbeater.cc:511] Master 127.14.33.62:41985 requested a full tablet report, sending...
I20260925 05:09:19.187144 31968 ts_manager.cc:194] Registered new tserver with Master: 6552d72f377c46f0ad938ac672612a13 (127.14.33.3:37573)
I20260925 05:09:19.187472 31968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.3:53699
I20260925 05:09:19.192788 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:19.196470 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.194042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.14.33.62:41985)
0925 05:09:19.194091 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.194092 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.194195 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:09:19.194256 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.194257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.194264 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.194302 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:19.194434 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.194872 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.195513 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.195676 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.195720 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.196304 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:19.196305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:19.196305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:19.196309 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:19.196319 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:19.196322 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:19.196358 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.196403 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.196418 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:09:19.197742 31968 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:19.199164 31968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:19.205021 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.201182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.14.33.3:37573)
0925 05:09:19.201330 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.201334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.201549 (+ 215us) server_negotiation.cc:407] Connection header received
0925 05:09:19.201602 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.201604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.201618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.201692 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:19.201844 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.202563 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.203968 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.204640 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.204716 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.204795 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.204909 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.204947 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":110,"threads_started":1}
I20260925 05:09:19.205050 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.201352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39187 (local address 127.0.0.1:45782)
0925 05:09:19.201464 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.201474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.201535 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.201933 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.201934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.201935 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:19.201969 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.201970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.203050 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.203051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.203856 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.203858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.204729 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.204730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.204806 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.204819 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.204893 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.204950 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260925 05:09:19.205056 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.200945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:37573 (local address 127.0.0.1:59404)
0925 05:09:19.201463 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.201474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.201535 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.201695 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.201696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.201697 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:19.201746 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.201748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.202639 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.202639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.203895 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.203897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.204727 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.204729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.204821 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.204823 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.204893 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.204951 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":33,"thread_start_us":60,"threads_started":1}
I20260925 05:09:19.205586 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.201280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.14.33.1:39187)
0925 05:09:19.201431 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.201435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.201738 (+ 303us) server_negotiation.cc:407] Connection header received
0925 05:09:19.201778 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.201780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.201795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.201862 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:19.202040 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.202971 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.203938 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.204640 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.204716 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.204796 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.205465 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.205507 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260925 05:09:19.205721 32352 tablet_service.cc:1467] Processing CreateTablet for tablet e383bd7cde854f65b5eb0124ad76b0bb (DEFAULT_TABLE table=test-table [id=08248d92b0b741338bc0d0dac10e7d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:19.205900 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.200945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44299 (local address 127.0.0.1:44298)
0925 05:09:19.201056 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.201097 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.201189 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.201443 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.201446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.201449 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:19.201505 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.201506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.202771 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.202771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.203895 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.203897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.205654 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.205655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.205716 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.205720 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.205793 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.205831 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":7,"thread_start_us":85,"threads_started":1}
I20260925 05:09:19.206058 32352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e383bd7cde854f65b5eb0124ad76b0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:19.206293 32090 tablet_service.cc:1467] Processing CreateTablet for tablet e383bd7cde854f65b5eb0124ad76b0bb (DEFAULT_TABLE table=test-table [id=08248d92b0b741338bc0d0dac10e7d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:19.206580 32090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e383bd7cde854f65b5eb0124ad76b0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:19.208524 32432 tablet_bootstrap.cc:492] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Bootstrap starting.
I20260925 05:09:19.209074 32433 tablet_bootstrap.cc:492] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732: Bootstrap starting.
I20260925 05:09:19.209214 32432 tablet_bootstrap.cc:654] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:19.209568 32432 log.cc:824] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:19.209620 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.201032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.14.33.2:44299)
0925 05:09:19.201247 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.201252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.201260 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:19.201301 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.201303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.201319 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.201586 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:19.201646 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.202692 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.204832 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.205569 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.205648 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.205735 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.209470 (+ 3735us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.209522 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260925 05:09:19.209705 32433 tablet_bootstrap.cc:654] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:19.210011 32433 log.cc:824] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:19.210314 32432 tablet_bootstrap.cc:492] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: No bootstrap required, opened a new log
I20260925 05:09:19.210219 32221 tablet_service.cc:1467] Processing CreateTablet for tablet e383bd7cde854f65b5eb0124ad76b0bb (DEFAULT_TABLE table=test-table [id=08248d92b0b741338bc0d0dac10e7d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:19.210391 32432 ts_tablet_manager.cc:1397] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:09:19.210523 32221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e383bd7cde854f65b5eb0124ad76b0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:19.210791 32433 tablet_bootstrap.cc:492] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732: No bootstrap required, opened a new log
I20260925 05:09:19.210877 32433 ts_tablet_manager.cc:1397] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:09:19.212968 32436 tablet_bootstrap.cc:492] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Bootstrap starting.
I20260925 05:09:19.212900 32432 raft_consensus.cc:348] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.213069 32432 raft_consensus.cc:374] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:19.213107 32432 raft_consensus.cc:729] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6552d72f377c46f0ad938ac672612a13, State: Initialized, Role: FOLLOWER
I20260925 05:09:19.213276 32432 consensus_queue.cc:260] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.213320 32433 raft_consensus.cc:348] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.213474 32433 raft_consensus.cc:374] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:19.213517 32433 raft_consensus.cc:729] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd53222e5c8247a5b182c85bb9a51732, State: Initialized, Role: FOLLOWER
I20260925 05:09:19.213552 32432 ts_tablet_manager.cc:1428] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 05:09:19.213667 32418 heartbeater.cc:503] Master 127.14.33.62:41985 was elected leader, sending a full tablet report...
I20260925 05:09:19.213650 32433 consensus_queue.cc:260] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.213743 32436 tablet_bootstrap.cc:654] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:19.213928 32433 ts_tablet_manager.cc:1428] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:09:19.213999 32156 heartbeater.cc:503] Master 127.14.33.62:41985 was elected leader, sending a full tablet report...
I20260925 05:09:19.214035 32436 log.cc:824] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:19.214769 32436 tablet_bootstrap.cc:492] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: No bootstrap required, opened a new log
I20260925 05:09:19.214866 32436 ts_tablet_manager.cc:1397] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:09:19.216784 32436 raft_consensus.cc:348] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.216912 32436 raft_consensus.cc:374] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:19.216933 32436 raft_consensus.cc:729] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4656836959ff4de0bac9604814cbb8d5, State: Initialized, Role: FOLLOWER
I20260925 05:09:19.217059 32436 consensus_queue.cc:260] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.217304 32436 ts_tablet_manager.cc:1428] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:09:19.217350 32287 heartbeater.cc:503] Master 127.14.33.62:41985 was elected leader, sending a full tablet report...
I20260925 05:09:19.293491 32437 raft_consensus.cc:482] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:19.293653 32437 raft_consensus.cc:504] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.294234 32437 leader_election.cc:288] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4656836959ff4de0bac9604814cbb8d5 (127.14.33.2:44299), fd53222e5c8247a5b182c85bb9a51732 (127.14.33.1:39187)
I20260925 05:09:19.297201 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.294467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:44299 (local address 127.14.33.3:54003)
0925 05:09:19.294531 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.294545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.294583 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.294781 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.294782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.294784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:19.294855 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.294856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.295606 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.295607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.296360 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.296362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.296914 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.296915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.296985 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.296988 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.297063 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.297104 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 05:09:19.297201 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.294512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:54003 (local address 127.14.33.2:44299)
0925 05:09:19.294559 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.294561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.294587 (+ 26us) server_negotiation.cc:407] Connection header received
0925 05:09:19.294686 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.294688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.294698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.294774 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:19.294928 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.295410 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.296456 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.296817 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.296912 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.296922 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.297072 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.297137 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:09:19.297403 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.294576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39187 (local address 127.14.33.3:38365)
0925 05:09:19.294751 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.294761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:19.294954 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:19.295124 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:19.295125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:19.295127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:19.295203 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.295206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.295827 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.295829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.296555 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:19.296557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.297104 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.297106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:19.297201 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.297205 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:19.297273 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 05:09:19.297317 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260925 05:09:19.297494 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.294687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:38365 (local address 127.14.33.1:39187)
0925 05:09:19.294741 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.294744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.294934 (+ 190us) server_negotiation.cc:407] Connection header received
0925 05:09:19.295034 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.295035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.295046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.295122 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:19.295285 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.295735 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.296650 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.297010 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.297093 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.297103 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.297369 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.297415 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:09:19.297506 32241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e383bd7cde854f65b5eb0124ad76b0bb" candidate_uuid: "6552d72f377c46f0ad938ac672612a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656836959ff4de0bac9604814cbb8d5" is_pre_election: true
I20260925 05:09:19.297659 32241 raft_consensus.cc:2454] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6552d72f377c46f0ad938ac672612a13 in term 0.
I20260925 05:09:19.297828 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e383bd7cde854f65b5eb0124ad76b0bb" candidate_uuid: "6552d72f377c46f0ad938ac672612a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd53222e5c8247a5b182c85bb9a51732" is_pre_election: true
I20260925 05:09:19.297912 32305 leader_election.cc:302] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4656836959ff4de0bac9604814cbb8d5, 6552d72f377c46f0ad938ac672612a13; no voters:
I20260925 05:09:19.298043 32437 raft_consensus.cc:2781] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:19.298100 32437 raft_consensus.cc:482] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:19.298117 32437 raft_consensus.cc:3037] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:19.298163 32109 raft_consensus.cc:2454] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6552d72f377c46f0ad938ac672612a13 in term 0.
I20260925 05:09:19.298902 32437 raft_consensus.cc:504] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.299031 32437 leader_election.cc:288] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [CANDIDATE]: Term 1 election: Requested vote from peers 4656836959ff4de0bac9604814cbb8d5 (127.14.33.2:44299), fd53222e5c8247a5b182c85bb9a51732 (127.14.33.1:39187)
I20260925 05:09:19.299186 32109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e383bd7cde854f65b5eb0124ad76b0bb" candidate_uuid: "6552d72f377c46f0ad938ac672612a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd53222e5c8247a5b182c85bb9a51732"
I20260925 05:09:19.299185 32241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e383bd7cde854f65b5eb0124ad76b0bb" candidate_uuid: "6552d72f377c46f0ad938ac672612a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4656836959ff4de0bac9604814cbb8d5"
I20260925 05:09:19.299265 32241 raft_consensus.cc:3037] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:19.299264 32109 raft_consensus.cc:3037] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:19.300158 32109 raft_consensus.cc:2454] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6552d72f377c46f0ad938ac672612a13 in term 1.
I20260925 05:09:19.300158 32241 raft_consensus.cc:2454] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6552d72f377c46f0ad938ac672612a13 in term 1.
I20260925 05:09:19.300423 32305 leader_election.cc:302] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6552d72f377c46f0ad938ac672612a13, fd53222e5c8247a5b182c85bb9a51732; no voters:
I20260925 05:09:19.300524 32437 raft_consensus.cc:2781] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:19.300681 32437 raft_consensus.cc:686] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 1 LEADER]: Becoming Leader. State: Replica: 6552d72f377c46f0ad938ac672612a13, State: Running, Role: LEADER
I20260925 05:09:19.300791 32437 consensus_queue.cc:237] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } }
I20260925 05:09:19.301577 31968 catalog_manager.cc:5495] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6552d72f377c46f0ad938ac672612a13 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "6552d72f377c46f0ad938ac672612a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552d72f377c46f0ad938ac672612a13" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 37573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 } health_report { overall_health: UNKNOWN } } }
W20260925 05:09:19.305492 32288 tablet.cc:2369] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:19.321659 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.319282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.14.33.3:37573)
0925 05:09:19.319388 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:19.319390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:19.319398 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:19.319493 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:19.319494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:19.319511 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:19.319579 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:19.319700 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.320133 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.320976 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:19.321220 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:19.321273 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:19.321405 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:19.321493 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:19.321553 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:19.321580 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 05:09:19.321593 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:09:19.323386 32109 raft_consensus.cc:1261] T e383bd7cde854f65b5eb0124ad76b0bb P fd53222e5c8247a5b182c85bb9a51732 [term 1 FOLLOWER]: Refusing update from remote peer 6552d72f377c46f0ad938ac672612a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:19.323385 32241 raft_consensus.cc:1261] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 1 FOLLOWER]: Refusing update from remote peer 6552d72f377c46f0ad938ac672612a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:19.323748 32442 consensus_queue.cc:1035] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd53222e5c8247a5b182c85bb9a51732" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 39187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:19.323869 32437 consensus_queue.cc:1035] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4656836959ff4de0bac9604814cbb8d5" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:19.328660 31968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:19.328763 31968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:19.329507 31968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13a6b619e06946e289096cc154cfa07b: Sending DeleteTablet for 3 replicas of tablet e383bd7cde854f65b5eb0124ad76b0bb
I20260925 05:09:19.329821 32221 tablet_service.cc:1514] Processing DeleteTablet for tablet e383bd7cde854f65b5eb0124ad76b0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:19 UTC) from {username='slave'} at 127.0.0.1:44298
I20260925 05:09:19.329873 32090 tablet_service.cc:1514] Processing DeleteTablet for tablet e383bd7cde854f65b5eb0124ad76b0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:19 UTC) from {username='slave'} at 127.0.0.1:45782
I20260925 05:09:19.330018 32455 tablet_replica.cc:330] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: stopping tablet replica
I20260925 05:09:19.330106 32455 raft_consensus.cc:2229] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:19.330159 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 32028
I20260925 05:09:19.330195 32455 raft_consensus.cc:2258] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:19.331143 32455 ts_tablet_manager.cc:1905] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:19.332545 32455 ts_tablet_manager.cc:1918] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:19.332620 32455 log.cc:1196] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/e383bd7cde854f65b5eb0124ad76b0bb
I20260925 05:09:19.332904 32455 ts_tablet_manager.cc:1939] T e383bd7cde854f65b5eb0124ad76b0bb P 4656836959ff4de0bac9604814cbb8d5: Deleting consensus metadata
I20260925 05:09:19.333320 31952 catalog_manager.cc:4841] TS 4656836959ff4de0bac9604814cbb8d5 (127.14.33.2:44299): tablet e383bd7cde854f65b5eb0124ad76b0bb (table test-table [id=08248d92b0b741338bc0d0dac10e7d7e]) successfully deleted
I20260925 05:09:19.333938 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet e383bd7cde854f65b5eb0124ad76b0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:19 UTC) from {username='slave'} at 127.0.0.1:59404
I20260925 05:09:19.334177 32457 tablet_replica.cc:330] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: stopping tablet replica
I20260925 05:09:19.334283 32457 raft_consensus.cc:2229] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:09:19.334393 32457 raft_consensus.cc:2258] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:19.334785 32457 ts_tablet_manager.cc:1905] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:19.336328 32457 ts_tablet_manager.cc:1918] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:19.336407 32457 log.cc:1196] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/e383bd7cde854f65b5eb0124ad76b0bb
I20260925 05:09:19.336632 32457 ts_tablet_manager.cc:1939] T e383bd7cde854f65b5eb0124ad76b0bb P 6552d72f377c46f0ad938ac672612a13: Deleting consensus metadata
I20260925 05:09:19.337002 31954 catalog_manager.cc:4841] TS 6552d72f377c46f0ad938ac672612a13 (127.14.33.3:37573): tablet e383bd7cde854f65b5eb0124ad76b0bb (table test-table [id=08248d92b0b741338bc0d0dac10e7d7e]) successfully deleted
W20260925 05:09:19.338106 31952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:39187 (error 108)
I20260925 05:09:19.338354 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 32159
I20260925 05:09:19.338939 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:19.338428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:39187 (local address 127.0.0.1:45788)
0925 05:09:19.338860 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:19.338905 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
W20260925 05:09:19.339145 31952 catalog_manager.cc:4584] TS fd53222e5c8247a5b182c85bb9a51732 (127.14.33.1:39187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e383bd7cde854f65b5eb0124ad76b0bb: Network error: Client connection negotiation failed: client connection to 127.14.33.1:39187: connect: Connection refused (error 111)
I20260925 05:09:19.345335 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 32290
I20260925 05:09:19.349745 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 31937
2026-09-25T05:09:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (711 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:19 dist-test-slave-4dmq krb5kdc[32461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:19 dist-test-slave-4dmq krb5kdc[32461](info): set up 2 sockets
Sep 25 05:09:19 dist-test-slave-4dmq krb5kdc[32461](info): commencing operation
krb5kdc: starting...
W20260925 05:09:20.349953 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:20.459259 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:46233
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:20.538264 32474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:20.538502 32474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:20.538544 32474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:20.539654 32474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:20.539721 32474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:20.539759 32474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:20.539793 32474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:20.539819 32474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:20.541258 32474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:46233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:20.541507 32474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:20.541746 32474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:20.546205 32479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:20.546232 32482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:20.546288 32480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:20.546279 32474 server_base.cc:1034] running on GCE node
I20260925 05:09:20.546630 32474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:20.546838 32474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:20.548015 32474 hybrid_clock.cc:648] HybridClock initialized: now 1790312960548002 us; error 34 us; skew 500 ppm
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:20.550439 32474 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:20.550827 32474 webserver.cc:460] Webserver started at http://127.14.33.62:42519/ using document root <none> and password file <none>
I20260925 05:09:20.551033 32474 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:20.551096 32474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:20.551236 32474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:20.552158 32474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "b7fb2ad1e0c84373b496a37d1b5f2216"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "1c278c91a624122f65158327b1ee6507"
server_key_iv: "6a9cd41c009bfb395fe197a172cb8000"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.552497 32474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "b7fb2ad1e0c84373b496a37d1b5f2216"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "1c278c91a624122f65158327b1ee6507"
server_key_iv: "6a9cd41c009bfb395fe197a172cb8000"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.553855 32474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:20.554509 32489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.554667 32474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:20.554761 32474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "b7fb2ad1e0c84373b496a37d1b5f2216"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "1c278c91a624122f65158327b1ee6507"
server_key_iv: "6a9cd41c009bfb395fe197a172cb8000"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.554833 32474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:20.593479 32474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:20.595767 32474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:20.595928 32474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:20.599675 32474 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:46233
I20260925 05:09:20.599713 32541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:46233 every 8 connection(s)
I20260925 05:09:20.600072 32474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:20.600940 32542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:20.603169 32542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: Bootstrap starting.
I20260925 05:09:20.603528 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 32474
I20260925 05:09:20.603655 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:20.603828 32542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:20.603839 14468 external_mini_cluster.cc:1428] Setting key 360da6bb8c0e38054f3fa90d9bc44f2d
I20260925 05:09:20.604166 32542 log.cc:824] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:20.605145 32542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: No bootstrap required, opened a new log
I20260925 05:09:20.607430 32542 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46233 } }
I20260925 05:09:20.607601 32542 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:20.607628 32542 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fb2ad1e0c84373b496a37d1b5f2216, State: Initialized, Role: FOLLOWER
I20260925 05:09:20.607770 32542 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46233 } }
I20260925 05:09:20.607846 32542 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:20.607868 32542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:20.607933 32542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:20.608537 32542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46233 } }
I20260925 05:09:20.608650 32542 leader_election.cc:302] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7fb2ad1e0c84373b496a37d1b5f2216; no voters:
I20260925 05:09:20.608803 32542 leader_election.cc:288] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:20.608865 32547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:20.609053 32547 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [term 1 LEADER]: Becoming Leader. State: Replica: b7fb2ad1e0c84373b496a37d1b5f2216, State: Running, Role: LEADER
I20260925 05:09:20.609180 32547 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46233 } }
I20260925 05:09:20.609046 32542 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:20.609561 32548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46233 } } }
I20260925 05:09:20.609665 32548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:20.610874 32549 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7fb2ad1e0c84373b496a37d1b5f2216. Latest consensus state: current_term: 1 leader_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fb2ad1e0c84373b496a37d1b5f2216" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 46233 } } }
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:20.610958 32549 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216 [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:20.611436 32562 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:20.611745 32562 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:20.611831 32551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:20.612073 32551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:20.614064 32551 catalog_manager.cc:1347] Generated new cluster ID: aa5e8e98b8644a51a63b868b4699062b
I20260925 05:09:20.614112 32551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:20.615674 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:20.604352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.14.33.62:46233)
0925 05:09:20.604521 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:20.604531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:20.604566 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:09:20.605139 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:20.605147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:20.605209 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:20.605360 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:20.605934 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.607404 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.608334 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.608977 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.609417 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:20.612048 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:20.612052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:20.612052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:20.612063 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:20.613965 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:20.614355 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:20.614358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:20.614358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:20.614452 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:20.614668 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:20.614668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:20.614669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:20.614968 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:20.615112 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:20.615217 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 05:09:20.615456 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260925 05:09:20.623943 32551 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:20.624454 32551 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:20.627427 32551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: Generated new TSK 0
I20260925 05:09:20.627592 32551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:09:20.653458 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46233
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:20.753003 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:20.753198 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:20.753221 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:20.754391 32568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:20.754431 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:20.754458 32568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:20.755932 32568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:20.756151 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:20.756399 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:20.761282 32573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:20.761286 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:20.761307 32576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:20.761693 32568 server_base.cc:1034] running on GCE node
I20260925 05:09:20.761878 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:20.762120 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:20.763257 32568 hybrid_clock.cc:648] HybridClock initialized: now 1790312960763236 us; error 37 us; skew 500 ppm
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:20.765645 32568 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:09:20.766058 32568 webserver.cc:460] Webserver started at http://127.14.33.1:37779/ using document root <none> and password file <none>
I20260925 05:09:20.766279 32568 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:20.766342 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:20.766486 32568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:20.767432 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "9410c0423e75403eb3e598ce1277aaf3"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "3b62d75d8604945e71565970c62d939b"
server_key_iv: "e3f135607edb251d9bfb2a638f0474a5"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.767786 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "9410c0423e75403eb3e598ce1277aaf3"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "3b62d75d8604945e71565970c62d939b"
server_key_iv: "e3f135607edb251d9bfb2a638f0474a5"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.769085 32568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:20.769783 32583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.769961 32568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:20.770051 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "9410c0423e75403eb3e598ce1277aaf3"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "3b62d75d8604945e71565970c62d939b"
server_key_iv: "e3f135607edb251d9bfb2a638f0474a5"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.770117 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:20.786624 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:20.788664 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:20.788812 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:20.789016 32568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:20.789332 32568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:20.789384 32568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.789415 32568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:20.789459 32568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.795042 32568 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:43145
I20260925 05:09:20.795086 32696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:43145 every 8 connection(s)
I20260925 05:09:20.795351 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:20.798211 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 32568
I20260925 05:09:20.798350 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:20.798529 14468 external_mini_cluster.cc:1428] Setting key 1148fd77ac2ebe745b7c735aec07b9b1
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:20.804787 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:20.795987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:49587 (local address 127.14.33.62:46233)
0925 05:09:20.796060 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:20.796062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:20.796190 (+ 128us) server_negotiation.cc:407] Connection header received
0925 05:09:20.796642 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:20.796643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:20.796652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:20.796700 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:20.797527 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.797996 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.799297 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.799622 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.799706 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:20.802044 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:20.802045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:20.802046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:20.802053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:20.803456 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:20.803880 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:20.803881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:20.803881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:20.803934 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:20.804183 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:20.804184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:20.804184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:20.804292 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:20.804358 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:20.804623 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 05:09:20.804700 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:09:20.804948 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:20.795939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46233 (local address 127.14.33.1:49587)
0925 05:09:20.796083 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:20.796115 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:20.796554 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:20.796746 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:20.796748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:20.797282 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:20.797453 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:20.797457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.798093 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.798093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:20.799189 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:20.799192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.799709 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.799709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:20.799802 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:20.800298 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:20.800310 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:20.801948 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:20.803566 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:20.803573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:20.803575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:20.803800 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:20.804005 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:20.804005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:20.804006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:20.804113 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:20.804361 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:20.804368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:20.804498 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 05:09:20.804592 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:09:20.804733 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20260925 05:09:20.805415 32697 heartbeater.cc:347] Connected to a master server at 127.14.33.62:46233
I20260925 05:09:20.805498 32697 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:20.805676 32697 heartbeater.cc:511] Master 127.14.33.62:46233 requested a full tablet report, sending...
I20260925 05:09:20.806160 32506 ts_manager.cc:194] Registered new tserver with Master: 9410c0423e75403eb3e598ce1277aaf3 (127.14.33.1:43145)
I20260925 05:09:20.806733 32506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:49587
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:09:20.821918 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46233
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:20.898871 32702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:20.899048 32702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:20.899071 32702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:20.900110 32702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:20.900144 32702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:20.900169 32702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:20.901521 32702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:20.901698 32702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:20.901863 32702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:20.906656 32710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:20.906674 32707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:20.906684 32708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:20.906821 32702 server_base.cc:1034] running on GCE node
I20260925 05:09:20.907060 32702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:20.907287 32702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:20.908439 32702 hybrid_clock.cc:648] HybridClock initialized: now 1790312960908428 us; error 28 us; skew 500 ppm
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:20.910990 32702 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:09:20.911437 32702 webserver.cc:460] Webserver started at http://127.14.33.2:33211/ using document root <none> and password file <none>
I20260925 05:09:20.911648 32702 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:20.911712 32702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:20.911865 32702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:20.912863 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "e35ee0c404624eabb5f9f2dad8a29ed9"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "edfcce29083fa6d3dc68dc2c1d43371f"
server_key_iv: "bb3a05bb85eb6becb58a729c623b68a5"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.913246 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "e35ee0c404624eabb5f9f2dad8a29ed9"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "edfcce29083fa6d3dc68dc2c1d43371f"
server_key_iv: "bb3a05bb85eb6becb58a729c623b68a5"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.914465 32702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:20.915100 32717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.915244 32702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:20.915336 32702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "e35ee0c404624eabb5f9f2dad8a29ed9"
format_stamp: "Formatted at 2026-09-25 05:09:20 on dist-test-slave-4dmq"
server_key: "edfcce29083fa6d3dc68dc2c1d43371f"
server_key_iv: "bb3a05bb85eb6becb58a729c623b68a5"
server_key_version: "encryptionkey@0"
I20260925 05:09:20.915423 32702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:20.923633 32702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:20.925346 32702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:20.925493 32702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:20.925688 32702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:20.925982 32702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:20.926033 32702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.926066 32702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:20.926093 32702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:20.931710 32702 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:40445
I20260925 05:09:20.931759 362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:40445 every 8 connection(s)
I20260925 05:09:20.932044 32702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:20.935988 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 32702
I20260925 05:09:20.936120 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:20.936319 14468 external_mini_cluster.cc:1428] Setting key c7d6e40322158cf9f642f60637691d35
Sep 25 05:09:20 dist-test-slave-4dmq krb5kdc[32461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312960, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:20.941591 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:20.932674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:44879 (local address 127.14.33.62:46233)
0925 05:09:20.932741 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:20.932742 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:20.933053 (+ 311us) server_negotiation.cc:407] Connection header received
0925 05:09:20.933600 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:20.933601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:20.933612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:20.933674 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:20.934575 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.935072 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.936044 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.936332 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.936421 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:20.938662 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:20.938663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:20.938664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:20.938670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:20.940287 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:20.940679 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:20.940680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:20.940681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:20.940737 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:20.940971 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:20.940973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:20.940973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:20.941082 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:20.941173 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:20.941462 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 05:09:20.941528 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 05:09:20.941865 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:20.932667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46233 (local address 127.14.33.2:44879)
0925 05:09:20.932831 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:20.932863 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:20.933495 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:20.933744 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:20.933747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:20.934290 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:20.934471 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:20.934475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.935169 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.935170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:20.935948 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:20.935950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:20.936432 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:20.936433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:20.936552 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:20.937090 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:20.937100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:20.938566 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:20.940364 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:20.940371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:20.940373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:20.940576 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:20.940798 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:20.940799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:20.940800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:20.940904 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:20.941163 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:20.941169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:20.941318 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 05:09:20.941431 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 05:09:20.941603 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260925 05:09:20.942360 363 heartbeater.cc:347] Connected to a master server at 127.14.33.62:46233
I20260925 05:09:20.942458 363 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:20.942647 363 heartbeater.cc:511] Master 127.14.33.62:46233 requested a full tablet report, sending...
I20260925 05:09:20.943039 32506 ts_manager.cc:194] Registered new tserver with Master: e35ee0c404624eabb5f9f2dad8a29ed9 (127.14.33.2:40445)
I20260925 05:09:20.943503 32506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:44879
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:09:20.959836 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46233
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:21.035459 368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:21.035692 368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:21.035730 368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:21.036827 368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:21.036895 368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:21.036939 368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:21.038408 368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:21.038669 368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:21.038880 368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:21.043782 374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.043821 373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.043799 376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.043992 368 server_base.cc:1034] running on GCE node
I20260925 05:09:21.044149 368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:21.044322 368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:21.045444 368 hybrid_clock.cc:648] HybridClock initialized: now 1790312961045434 us; error 27 us; skew 500 ppm
Sep 25 05:09:21 dist-test-slave-4dmq krb5kdc[32461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312961, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:21.047734 368 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:09:21.048151 368 webserver.cc:460] Webserver started at http://127.14.33.3:37851/ using document root <none> and password file <none>
I20260925 05:09:21.048362 368 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:21.048412 368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:21.048548 368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:21.049600 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "fdbbb2f8fc0b46da9a069788f76a3a25"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "b4e9b43769f25373de01a22a6505ad93"
server_key_iv: "190b68f2e397e28a42986c1607f71943"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.049914 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "fdbbb2f8fc0b46da9a069788f76a3a25"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "b4e9b43769f25373de01a22a6505ad93"
server_key_iv: "190b68f2e397e28a42986c1607f71943"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.051118 368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:21.051784 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.051965 368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:21.052075 368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "fdbbb2f8fc0b46da9a069788f76a3a25"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "b4e9b43769f25373de01a22a6505ad93"
server_key_iv: "190b68f2e397e28a42986c1607f71943"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.052137 368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:21.062456 368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:21.064073 368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:21.064214 368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:21.064409 368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:21.064697 368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:21.064747 368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.064786 368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:21.064814 368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.070415 368 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:44117
I20260925 05:09:21.070451 496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:44117 every 8 connection(s)
I20260925 05:09:21.070736 368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:09:21.073772 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 368
I20260925 05:09:21.073900 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:21.074079 14468 external_mini_cluster.cc:1428] Setting key 9ec39e1d43d87959f42b88004f2f87b9
Sep 25 05:09:21 dist-test-slave-4dmq krb5kdc[32461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312961, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:21.079348 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.071353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:51133 (local address 127.14.33.62:46233)
0925 05:09:21.071391 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.071394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.071587 (+ 193us) server_negotiation.cc:407] Connection header received
0925 05:09:21.072021 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.072022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.072032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.072080 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.072968 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.073490 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.074901 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.075116 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.075167 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.077092 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.077093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.077093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:21.077098 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.078282 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:21.078607 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:21.078607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:21.078607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:21.078652 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:21.078881 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:21.078881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:21.078882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:21.078981 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.079024 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.079216 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.079289 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:09:21.079514 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.071315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:46233 (local address 127.14.33.3:51133)
0925 05:09:21.071437 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.071511 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.071944 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.072150 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.072152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.072650 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:21.072890 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.072896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.073561 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.073562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.074787 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.074789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.075181 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.075182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.075281 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.075758 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:21.075765 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:21.077001 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:21.078347 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:21.078353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:21.078354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:21.078548 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:21.078715 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:21.078715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:21.078715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:21.078825 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:21.079028 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:21.079032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:21.079124 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.079195 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.079326 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260925 05:09:21.079977 497 heartbeater.cc:347] Connected to a master server at 127.14.33.62:46233
I20260925 05:09:21.080058 497 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:21.080179 497 heartbeater.cc:511] Master 127.14.33.62:46233 requested a full tablet report, sending...
I20260925 05:09:21.080487 32506 ts_manager.cc:194] Registered new tserver with Master: fdbbb2f8fc0b46da9a069788f76a3a25 (127.14.33.3:44117)
I20260925 05:09:21.080827 32506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:51133
I20260925 05:09:21.085651 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:21.092442 32545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.086841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.14.33.62:46233)
0925 05:09:21.086927 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.086929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.086972 (+ 43us) server_negotiation.cc:407] Connection header received
0925 05:09:21.087033 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.087034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.087040 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.087090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.087592 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.088014 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.088660 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.088836 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.088888 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.090106 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.090107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.090107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:21.090113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.091356 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:21.091787 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:21.091787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:21.091787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:21.091832 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:21.092000 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:21.092000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:21.092001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:21.092105 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.092179 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.092283 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.092324 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:09:21.093858 32506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:21.095337 32506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:21.100986 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.097283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:40445 (local address 127.0.0.1:56816)
0925 05:09:21.097657 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.097668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.097716 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.098049 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.098049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.098051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.098079 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.098080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.099199 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.099200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.099921 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.099923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.100772 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.100773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.100840 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.100850 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.100900 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.100935 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":7,"thread_start_us":71,"threads_started":1}
I20260925 05:09:21.101038 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.097499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.14.33.2:40445)
0925 05:09:21.097879 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.097883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.097891 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:21.097927 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.097929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.097943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.098044 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.098129 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.099110 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.100025 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.100700 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.100767 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.100816 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.100913 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.100946 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":65,"threads_started":1}
I20260925 05:09:21.101723 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.097277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:44117 (local address 127.0.0.1:38410)
0925 05:09:21.097385 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.097420 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.097480 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.097755 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.097757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.097759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.097818 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.097820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.099094 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.099094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.100329 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.100331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.101509 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.101510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.101571 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.101574 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.101621 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.101656 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":87,"thread_start_us":62,"threads_started":1}
I20260925 05:09:21.102011 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.097498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.14.33.3:44117)
0925 05:09:21.097627 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.097630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.097636 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:09:21.097660 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.097661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.097671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.097751 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.097902 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.099001 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.100386 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.100988 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.101052 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.101100 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.101894 (+ 794us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.101942 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 05:09:21.102008 32765 tablet_service.cc:1467] Processing CreateTablet for tablet 155138359d4e46a28b33bef335ed4d40 (DEFAULT_TABLE table=test-table [id=0f6add81abab417d97018f6192e48c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:21.102200 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.097283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:43145 (local address 127.0.0.1:33086)
0925 05:09:21.097418 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.097426 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.097480 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.097959 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.097961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.097963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.098003 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.098004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.099015 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.099016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.100315 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.100318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.101967 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.101968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.102030 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.102035 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.102094 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.102132 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":63,"thread_start_us":60,"threads_started":1}
I20260925 05:09:21.102377 32765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155138359d4e46a28b33bef335ed4d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.102488 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.097651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.14.33.1:43145)
0925 05:09:21.097811 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.097815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.097823 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:21.097862 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.097863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.097873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.097951 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.098164 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.098915 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.101151 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.101867 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.101956 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.102011 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.102368 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.102413 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260925 05:09:21.103015 32631 tablet_service.cc:1467] Processing CreateTablet for tablet 155138359d4e46a28b33bef335ed4d40 (DEFAULT_TABLE table=test-table [id=0f6add81abab417d97018f6192e48c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:21.103291 32631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155138359d4e46a28b33bef335ed4d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.105033 511 tablet_bootstrap.cc:492] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Bootstrap starting.
I20260925 05:09:21.105451 431 tablet_service.cc:1467] Processing CreateTablet for tablet 155138359d4e46a28b33bef335ed4d40 (DEFAULT_TABLE table=test-table [id=0f6add81abab417d97018f6192e48c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:21.105669 511 tablet_bootstrap.cc:654] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.105747 431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155138359d4e46a28b33bef335ed4d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.105865 512 tablet_bootstrap.cc:492] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3: Bootstrap starting.
I20260925 05:09:21.106212 511 log.cc:824] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.106595 512 tablet_bootstrap.cc:654] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.107038 511 tablet_bootstrap.cc:492] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: No bootstrap required, opened a new log
I20260925 05:09:21.107065 512 log.cc:824] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.107115 511 ts_tablet_manager.cc:1397] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:09:21.107690 512 tablet_bootstrap.cc:492] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3: No bootstrap required, opened a new log
I20260925 05:09:21.107762 512 ts_tablet_manager.cc:1397] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:21.108848 515 tablet_bootstrap.cc:492] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Bootstrap starting.
I20260925 05:09:21.109050 511 raft_consensus.cc:348] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.109318 511 raft_consensus.cc:374] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.109367 511 raft_consensus.cc:729] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ee0c404624eabb5f9f2dad8a29ed9, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.109337 512 raft_consensus.cc:348] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.109501 512 raft_consensus.cc:374] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.109504 511 consensus_queue.cc:260] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.109550 512 raft_consensus.cc:729] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9410c0423e75403eb3e598ce1277aaf3, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.109714 512 consensus_queue.cc:260] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.109817 515 tablet_bootstrap.cc:654] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.109822 363 heartbeater.cc:503] Master 127.14.33.62:46233 was elected leader, sending a full tablet report...
I20260925 05:09:21.110059 32697 heartbeater.cc:503] Master 127.14.33.62:46233 was elected leader, sending a full tablet report...
I20260925 05:09:21.110177 511 ts_tablet_manager.cc:1428] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:09:21.110360 515 log.cc:824] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.110064 512 ts_tablet_manager.cc:1428] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:09:21.111222 515 tablet_bootstrap.cc:492] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: No bootstrap required, opened a new log
I20260925 05:09:21.111299 515 ts_tablet_manager.cc:1397] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 05:09:21.112478 515 raft_consensus.cc:348] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.112613 515 raft_consensus.cc:374] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.112653 515 raft_consensus.cc:729] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbbb2f8fc0b46da9a069788f76a3a25, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.112748 515 consensus_queue.cc:260] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.112975 515 ts_tablet_manager.cc:1428] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:09:21.112977 497 heartbeater.cc:503] Master 127.14.33.62:46233 was elected leader, sending a full tablet report...
I20260925 05:09:21.165993 517 raft_consensus.cc:482] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:21.166128 517 raft_consensus.cc:504] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.166468 517 leader_election.cc:288] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdbbb2f8fc0b46da9a069788f76a3a25 (127.14.33.3:44117), e35ee0c404624eabb5f9f2dad8a29ed9 (127.14.33.2:40445)
I20260925 05:09:21.169320 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.166676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:47379 (local address 127.14.33.3:44117)
0925 05:09:21.166730 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.166732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.166900 (+ 168us) server_negotiation.cc:407] Connection header received
0925 05:09:21.166958 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.166959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.166971 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.167038 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.167174 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.167613 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.168547 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.168969 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.169042 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.169052 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.169214 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.169250 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:09:21.169322 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.166757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:47245 (local address 127.14.33.2:40445)
0925 05:09:21.166837 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.166839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.166848 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:09:21.166879 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.166880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.166889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.166963 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.167111 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.167613 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.168546 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.168926 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.169003 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.169014 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.169206 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.169241 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:09:21.169334 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.166611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:40445 (local address 127.14.33.1:47245)
0925 05:09:21.166649 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.166659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.166696 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.166969 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.166971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.166972 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.167040 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.167042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.167693 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.167694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.168454 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.168456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.169015 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.169016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.169098 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.169101 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.169195 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.169237 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":38}
I20260925 05:09:21.169353 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.166611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:44117 (local address 127.14.33.1:47379)
0925 05:09:21.166829 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.166836 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.166906 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.167041 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.167042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.167043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.167112 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.167113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.167695 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.167696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.168460 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.168461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.169044 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.169045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.169111 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.169149 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.169207 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.169247 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":56,"thread_start_us":91,"threads_started":1}
I20260925 05:09:21.169611 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155138359d4e46a28b33bef335ed4d40" candidate_uuid: "9410c0423e75403eb3e598ce1277aaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" is_pre_election: true
I20260925 05:09:21.169674 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155138359d4e46a28b33bef335ed4d40" candidate_uuid: "9410c0423e75403eb3e598ce1277aaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" is_pre_election: true
I20260925 05:09:21.169751 317 raft_consensus.cc:2454] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9410c0423e75403eb3e598ce1277aaf3 in term 0.
I20260925 05:09:21.169831 451 raft_consensus.cc:2454] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9410c0423e75403eb3e598ce1277aaf3 in term 0.
I20260925 05:09:21.169955 32584 leader_election.cc:302] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9410c0423e75403eb3e598ce1277aaf3, e35ee0c404624eabb5f9f2dad8a29ed9; no voters:
I20260925 05:09:21.170077 517 raft_consensus.cc:2781] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:21.170156 517 raft_consensus.cc:482] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:21.170181 517 raft_consensus.cc:3037] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:21.171017 517 raft_consensus.cc:504] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.171151 517 leader_election.cc:288] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [CANDIDATE]: Term 1 election: Requested vote from peers fdbbb2f8fc0b46da9a069788f76a3a25 (127.14.33.3:44117), e35ee0c404624eabb5f9f2dad8a29ed9 (127.14.33.2:40445)
I20260925 05:09:21.171288 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155138359d4e46a28b33bef335ed4d40" candidate_uuid: "9410c0423e75403eb3e598ce1277aaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25"
I20260925 05:09:21.171312 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155138359d4e46a28b33bef335ed4d40" candidate_uuid: "9410c0423e75403eb3e598ce1277aaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9"
I20260925 05:09:21.171363 451 raft_consensus.cc:3037] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:21.171372 317 raft_consensus.cc:3037] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:21.172251 317 raft_consensus.cc:2454] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9410c0423e75403eb3e598ce1277aaf3 in term 1.
I20260925 05:09:21.172251 451 raft_consensus.cc:2454] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9410c0423e75403eb3e598ce1277aaf3 in term 1.
I20260925 05:09:21.172421 32585 leader_election.cc:302] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9410c0423e75403eb3e598ce1277aaf3, fdbbb2f8fc0b46da9a069788f76a3a25; no voters:
I20260925 05:09:21.172539 517 raft_consensus.cc:2781] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:21.172675 517 raft_consensus.cc:686] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [term 1 LEADER]: Becoming Leader. State: Replica: 9410c0423e75403eb3e598ce1277aaf3, State: Running, Role: LEADER
I20260925 05:09:21.172772 517 consensus_queue.cc:237] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } }
I20260925 05:09:21.173578 32506 catalog_manager.cc:5495] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9410c0423e75403eb3e598ce1277aaf3 (127.14.33.1). New cstate: current_term: 1 leader_uuid: "9410c0423e75403eb3e598ce1277aaf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9410c0423e75403eb3e598ce1277aaf3" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 } health_report { overall_health: UNKNOWN } } }
W20260925 05:09:21.182512 364 tablet.cc:2369] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:21.192627 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.190448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.14.33.1:43145)
0925 05:09:21.190486 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.190489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.190497 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:21.190552 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.190553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.190577 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.190654 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:21.190781 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.191230 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.192005 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.192188 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.192240 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.192362 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:21.192441 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:21.192502 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.192572 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.192578 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:09:21.194398 451 raft_consensus.cc:1261] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 1 FOLLOWER]: Refusing update from remote peer 9410c0423e75403eb3e598ce1277aaf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:21.194413 317 raft_consensus.cc:1261] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 1 FOLLOWER]: Refusing update from remote peer 9410c0423e75403eb3e598ce1277aaf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:21.194766 517 consensus_queue.cc:1035] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdbbb2f8fc0b46da9a069788f76a3a25" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:21.194903 521 consensus_queue.cc:1035] T 155138359d4e46a28b33bef335ed4d40 P 9410c0423e75403eb3e598ce1277aaf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35ee0c404624eabb5f9f2dad8a29ed9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 40445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:21.196810 524 mvcc.cc:205] Tried to move back new op lower bound from 7333121889049452544 to 7333121888964247552. Current Snapshot: MvccSnapshot[applied={T|T < 7333121889049452544}]
I20260925 05:09:21.197221 525 mvcc.cc:205] Tried to move back new op lower bound from 7333121889049452544 to 7333121888964247552. Current Snapshot: MvccSnapshot[applied={T|T < 7333121889049452544}]
I20260925 05:09:21.197356 526 mvcc.cc:205] Tried to move back new op lower bound from 7333121889049452544 to 7333121888964247552. Current Snapshot: MvccSnapshot[applied={T|T < 7333121889049452544}]
I20260925 05:09:21.198781 32505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:21.198884 32505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:21.199595 32505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7fb2ad1e0c84373b496a37d1b5f2216: Sending DeleteTablet for 3 replicas of tablet 155138359d4e46a28b33bef335ed4d40
I20260925 05:09:21.199899 32631 tablet_service.cc:1514] Processing DeleteTablet for tablet 155138359d4e46a28b33bef335ed4d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:21 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:33086
I20260925 05:09:21.199913 32765 tablet_service.cc:1514] Processing DeleteTablet for tablet 155138359d4e46a28b33bef335ed4d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:21 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:56816
I20260925 05:09:21.200126 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 32568
I20260925 05:09:21.200151 534 tablet_replica.cc:330] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: stopping tablet replica
I20260925 05:09:21.200232 534 raft_consensus.cc:2229] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:21.200325 534 raft_consensus.cc:2258] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:21.200616 431 tablet_service.cc:1514] Processing DeleteTablet for tablet 155138359d4e46a28b33bef335ed4d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:21 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:38410
I20260925 05:09:21.200925 536 tablet_replica.cc:330] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: stopping tablet replica
I20260925 05:09:21.201009 536 raft_consensus.cc:2229] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:21.201054 534 ts_tablet_manager.cc:1905] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:21.201089 536 raft_consensus.cc:2258] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:21.201519 536 ts_tablet_manager.cc:1905] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:21.202641 534 ts_tablet_manager.cc:1918] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:21.202708 534 log.cc:1196] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/155138359d4e46a28b33bef335ed4d40
I20260925 05:09:21.202978 534 ts_tablet_manager.cc:1939] T 155138359d4e46a28b33bef335ed4d40 P e35ee0c404624eabb5f9f2dad8a29ed9: Deleting consensus metadata
I20260925 05:09:21.203198 536 ts_tablet_manager.cc:1918] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:21.203271 536 log.cc:1196] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/155138359d4e46a28b33bef335ed4d40
I20260925 05:09:21.203353 32490 catalog_manager.cc:4841] TS e35ee0c404624eabb5f9f2dad8a29ed9 (127.14.33.2:40445): tablet 155138359d4e46a28b33bef335ed4d40 (table test-table [id=0f6add81abab417d97018f6192e48c6e]) successfully deleted
I20260925 05:09:21.203547 536 ts_tablet_manager.cc:1939] T 155138359d4e46a28b33bef335ed4d40 P fdbbb2f8fc0b46da9a069788f76a3a25: Deleting consensus metadata
I20260925 05:09:21.203915 32491 catalog_manager.cc:4841] TS fdbbb2f8fc0b46da9a069788f76a3a25 (127.14.33.3:44117): tablet 155138359d4e46a28b33bef335ed4d40 (table test-table [id=0f6add81abab417d97018f6192e48c6e]) successfully deleted
W20260925 05:09:21.206043 32492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:43145 (error 108)
I20260925 05:09:21.206482 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 32702
I20260925 05:09:21.206497 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.206359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:43145 (local address 127.0.0.1:33102)
0925 05:09:21.206426 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.206466 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:43145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:09:21.206614 32492 catalog_manager.cc:4584] TS 9410c0423e75403eb3e598ce1277aaf3 (127.14.33.1:43145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 155138359d4e46a28b33bef335ed4d40: Network error: Client connection negotiation failed: client connection to 127.14.33.1:43145: connect: Connection refused (error 111)
I20260925 05:09:21.211175 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 368
I20260925 05:09:21.215523 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 32474
2026-09-25T05:09:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1866 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T05:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:21Z Disabled control of system clock
I20260925 05:09:21.243098 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38129
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:21.321903 541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:21.322100 541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:21.322120 541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:21.323243 541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:21.323287 541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:21.323299 541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:21.323338 541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:21.323369 541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:21.324697 541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:21.324949 541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:21.325265 541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:21.329872 546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.329890 549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.329890 547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.329970 541 server_base.cc:1034] running on GCE node
I20260925 05:09:21.330354 541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:21.330641 541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:21.331777 541 hybrid_clock.cc:648] HybridClock initialized: now 1790312961331756 us; error 45 us; skew 500 ppm
I20260925 05:09:21.332988 541 webserver.cc:460] Webserver started at http://127.14.33.62:42159/ using document root <none> and password file <none>
I20260925 05:09:21.333256 541 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:21.333302 541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:21.333393 541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:21.334348 541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "77c3a2b258063dad006e3bb94ef4a3e6"
server_key_iv: "9a885be4e00b38b08e6051b69acf1987"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.334632 541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "77c3a2b258063dad006e3bb94ef4a3e6"
server_key_iv: "9a885be4e00b38b08e6051b69acf1987"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.335918 541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:21.336648 555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.336933 541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:21.337042 541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "77c3a2b258063dad006e3bb94ef4a3e6"
server_key_iv: "9a885be4e00b38b08e6051b69acf1987"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.337148 541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:21.345500 541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:21.345777 541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:21.345921 541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:21.349694 541 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38129
I20260925 05:09:21.349745 607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38129 every 8 connection(s)
I20260925 05:09:21.350096 541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:21.351091 608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.353502 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: Bootstrap starting.
I20260925 05:09:21.354100 608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.354414 608 log.cc:824] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.355139 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: No bootstrap required, opened a new log
I20260925 05:09:21.356815 608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38129 } }
I20260925 05:09:21.357010 608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.357056 608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d2455e4b9c4b3ba8f3b92c5de2e060, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.357204 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 541
I20260925 05:09:21.357249 608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38129 } }
I20260925 05:09:21.357326 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:21.357353 608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:21.357398 608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:21.357458 608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:21.357559 14468 external_mini_cluster.cc:1428] Setting key 5de98898722c17872a44119364de89cc
I20260925 05:09:21.358220 608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38129 } }
I20260925 05:09:21.358345 608 leader_election.cc:302] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73d2455e4b9c4b3ba8f3b92c5de2e060; no voters:
I20260925 05:09:21.358556 608 leader_election.cc:288] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:21.358596 613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:21.358779 613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [term 1 LEADER]: Becoming Leader. State: Replica: 73d2455e4b9c4b3ba8f3b92c5de2e060, State: Running, Role: LEADER
I20260925 05:09:21.358822 608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:21.358947 613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38129 } }
I20260925 05:09:21.359404 614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38129 } } }
I20260925 05:09:21.359530 614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:21.359756 615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d2455e4b9c4b3ba8f3b92c5de2e060. Latest consensus state: current_term: 1 leader_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d2455e4b9c4b3ba8f3b92c5de2e060" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38129 } } }
I20260925 05:09:21.360091 615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060 [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:21.360805 629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:21.360867 629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:21.360919 622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:21.361078 622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:21.362766 622 catalog_manager.cc:1347] Generated new cluster ID: 6634540ae5f1446ba3e1b458cf9cdda2
I20260925 05:09:21.362823 622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:21.364589 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.358106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.14.33.62:38129)
0925 05:09:21.358338 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.358347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.358383 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:09:21.358528 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.358549 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.358631 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.359048 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.359156 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.361055 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.362328 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.362963 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.363096 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.363855 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.363859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.363859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:21.364042 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.364075 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:21.364087 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.364163 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.364231 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.364423 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260925 05:09:21.367405 622 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:21.368008 622 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:21.373351 622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: Generated new TSK 0
I20260925 05:09:21.373564 622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:21.381280 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38129
--builtin_ntp_servers=127.14.33.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:21.461930 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:21.462138 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:21.462174 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:21.463310 632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:21.463374 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:21.463418 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:21.464790 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:21.465057 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:21.465381 632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:21.470281 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.470321 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.470515 632 server_base.cc:1034] running on GCE node
W20260925 05:09:21.470289 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.470758 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:21.471022 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:21.472185 632 hybrid_clock.cc:648] HybridClock initialized: now 1790312961472159 us; error 38 us; skew 500 ppm
I20260925 05:09:21.473551 632 webserver.cc:460] Webserver started at http://127.14.33.1:33567/ using document root <none> and password file <none>
I20260925 05:09:21.473819 632 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:21.473886 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:21.474012 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:21.475346 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "954d55b299984a26b237cc2e6130ee00"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "aa23e1060a526719713188a355b5f3ba"
server_key_iv: "0819f3f3f3b5a6669b54a98e80cd4fd5"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.475745 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "954d55b299984a26b237cc2e6130ee00"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "aa23e1060a526719713188a355b5f3ba"
server_key_iv: "0819f3f3f3b5a6669b54a98e80cd4fd5"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.477380 632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:21.478209 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.478397 632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:21.478499 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "954d55b299984a26b237cc2e6130ee00"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "aa23e1060a526719713188a355b5f3ba"
server_key_iv: "0819f3f3f3b5a6669b54a98e80cd4fd5"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.478578 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:21.498893 632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:21.499207 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:21.499374 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:21.499610 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:21.499934 632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:21.499967 632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.499989 632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:21.500034 632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.505668 632 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:36887
I20260925 05:09:21.505764 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:36887 every 8 connection(s)
I20260925 05:09:21.506083 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:21.506371 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 632
I20260925 05:09:21.506493 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:21.506717 14468 external_mini_cluster.cc:1428] Setting key 8009cb2c20784d335b1ba2897f9fd990
I20260925 05:09:21.507638 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38129
--builtin_ntp_servers=127.14.33.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:09:21.511709 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.506841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:55141 (local address 127.14.33.62:38129)
0925 05:09:21.506921 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.506924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.507297 (+ 373us) server_negotiation.cc:407] Connection header received
0925 05:09:21.507908 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.507909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.507923 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.507972 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.508654 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.509252 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.510238 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.510460 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.510548 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.511415 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.511416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.511417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:21.511421 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.511437 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:21.511442 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.511494 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.511596 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.511629 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 05:09:21.511919 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.506840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38129 (local address 127.14.33.1:55141)
0925 05:09:21.507163 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.507205 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.507769 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.508050 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.508053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.508061 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:21.508559 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.508563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.509343 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.509344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.510142 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.510144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.510559 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.510560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.510691 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.511249 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:21.511249 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:21.511278 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:21.511301 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:21.511501 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:21.511506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:21.511509 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.511597 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.511697 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":233,"threads_started":1}
I20260925 05:09:21.512454 760 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38129
I20260925 05:09:21.512558 760 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:21.512729 760 heartbeater.cc:511] Master 127.14.33.62:38129 requested a full tablet report, sending...
I20260925 05:09:21.513334 572 ts_manager.cc:194] Registered new tserver with Master: 954d55b299984a26b237cc2e6130ee00 (127.14.33.1:36887)
I20260925 05:09:21.514319 572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.1:55141
W20260925 05:09:21.588557 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:21.588812 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:21.588838 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:21.590046 763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:21.590088 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:21.590113 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:21.591351 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:21.591522 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:21.591811 763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:21.596814 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.596818 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.596850 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.597030 763 server_base.cc:1034] running on GCE node
I20260925 05:09:21.597268 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:21.597478 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:21.598641 763 hybrid_clock.cc:648] HybridClock initialized: now 1790312961598628 us; error 31 us; skew 500 ppm
I20260925 05:09:21.599891 763 webserver.cc:460] Webserver started at http://127.14.33.2:36565/ using document root <none> and password file <none>
I20260925 05:09:21.600143 763 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:21.600216 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:21.600391 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:21.601552 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "fb1ab145149647319528d128fb809b66"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "add9e4c5348f9e114ed1c92cf6dea706"
server_key_iv: "70bb2975be3a79027f1ea30b7cf8e644"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.601940 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "fb1ab145149647319528d128fb809b66"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "add9e4c5348f9e114ed1c92cf6dea706"
server_key_iv: "70bb2975be3a79027f1ea30b7cf8e644"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.603228 763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:21.603971 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.604168 763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:21.604290 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "fb1ab145149647319528d128fb809b66"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "add9e4c5348f9e114ed1c92cf6dea706"
server_key_iv: "70bb2975be3a79027f1ea30b7cf8e644"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.604367 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:21.640969 763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:21.641336 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:21.641494 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:21.641731 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:21.642074 763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:21.642127 763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.642174 763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:21.642198 763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.648046 763 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:35099
I20260925 05:09:21.648103 893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:35099 every 8 connection(s)
I20260925 05:09:21.648481 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:21.652640 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 763
I20260925 05:09:21.652792 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:21.653041 14468 external_mini_cluster.cc:1428] Setting key 87f3ceef1ea5b43b64fbe306dcf48d2c
I20260925 05:09:21.653896 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.649326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:39053 (local address 127.14.33.62:38129)
0925 05:09:21.649388 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.649390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.649584 (+ 194us) server_negotiation.cc:407] Connection header received
0925 05:09:21.650069 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.650070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.650080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.650149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.650806 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.651323 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.652272 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.652547 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.652623 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.653617 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.653618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.653619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:21.653623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.653638 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:21.653643 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.653694 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.653788 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.653814 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 05:09:21.654000 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38129
--builtin_ntp_servers=127.14.33.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:09:21.654141 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.649279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38129 (local address 127.14.33.2:39053)
0925 05:09:21.649452 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.649493 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.649972 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.650204 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.650207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.650215 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:21.650700 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.650704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.651413 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.651414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.652188 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.652190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.652641 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.652642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.652788 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.653471 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:21.653471 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:21.653500 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:21.653522 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:21.653709 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:21.653714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:21.653716 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.653805 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.653909 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 05:09:21.654582 894 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38129
I20260925 05:09:21.654676 894 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:21.654852 894 heartbeater.cc:511] Master 127.14.33.62:38129 requested a full tablet report, sending...
I20260925 05:09:21.655232 572 ts_manager.cc:194] Registered new tserver with Master: fb1ab145149647319528d128fb809b66 (127.14.33.2:35099)
I20260925 05:09:21.655642 572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.2:39053
W20260925 05:09:21.733897 897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:21.734094 897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:21.734115 897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:21.735325 897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:21.735365 897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:21.735390 897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:21.736764 897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:21.737012 897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:21.737321 897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:21.742272 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.742412 897 server_base.cc:1034] running on GCE node
W20260925 05:09:21.742277 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:21.742288 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:21.742729 897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:21.743022 897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:21.744179 897 hybrid_clock.cc:648] HybridClock initialized: now 1790312961744175 us; error 36 us; skew 500 ppm
I20260925 05:09:21.745414 897 webserver.cc:460] Webserver started at http://127.14.33.3:45131/ using document root <none> and password file <none>
I20260925 05:09:21.745639 897 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:21.745704 897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:21.745853 897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:21.746898 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "52422d52240244b68ec5c2f5d8b5e185"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "ed996ae1715895d78b4077ca6e74e70d"
server_key_iv: "5942208fd3bf4d509787462e82fda2f1"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.747251 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "52422d52240244b68ec5c2f5d8b5e185"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "ed996ae1715895d78b4077ca6e74e70d"
server_key_iv: "5942208fd3bf4d509787462e82fda2f1"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.748467 897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:21.749068 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.749351 897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:21.749449 897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "52422d52240244b68ec5c2f5d8b5e185"
format_stamp: "Formatted at 2026-09-25 05:09:21 on dist-test-slave-4dmq"
server_key: "ed996ae1715895d78b4077ca6e74e70d"
server_key_iv: "5942208fd3bf4d509787462e82fda2f1"
server_key_version: "encryptionkey@0"
I20260925 05:09:21.749523 897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:21.762698 897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:21.762940 897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:21.763075 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:21.763293 897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:21.763608 897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:21.763672 897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.763710 897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:21.763739 897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:21.769268 897 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:38351
I20260925 05:09:21.769313 1024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:38351 every 8 connection(s)
I20260925 05:09:21.769654 897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:09:21.778138 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.770381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:51905 (local address 127.14.33.62:38129)
0925 05:09:21.770435 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.770437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.770670 (+ 233us) server_negotiation.cc:407] Connection header received
0925 05:09:21.771085 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.771086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.771097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.771159 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.771736 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.772211 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.773330 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.773520 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.773582 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.777875 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.777876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.777876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:21.777880 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.777896 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:21.777900 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.777945 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.778053 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.778079 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 05:09:21.778316 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.770352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:38129 (local address 127.14.33.3:51905)
0925 05:09:21.770516 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.770554 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.771005 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.771208 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.771211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.771218 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:21.771671 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.771675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.772303 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.772304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.773253 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.773256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.773594 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.773595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.773691 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.777749 (+ 4058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:21.777749 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:21.777776 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:21.777796 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:21.777954 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:21.777957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:21.777960 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.778028 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.778102 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 05:09:21.778825 1025 heartbeater.cc:347] Connected to a master server at 127.14.33.62:38129
I20260925 05:09:21.778927 1025 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:21.779002 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 897
I20260925 05:09:21.779121 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:21.779129 1025 heartbeater.cc:511] Master 127.14.33.62:38129 requested a full tablet report, sending...
I20260925 05:09:21.779381 14468 external_mini_cluster.cc:1428] Setting key c7b340cb5b72bffda16a5de0445ecd27
I20260925 05:09:21.779528 572 ts_manager.cc:194] Registered new tserver with Master: 52422d52240244b68ec5c2f5d8b5e185 (127.14.33.3:38351)
I20260925 05:09:21.779978 572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.3:51905
I20260925 05:09:21.780254 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:21.784137 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.781633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.14.33.62:38129)
0925 05:09:21.781667 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.781669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.781779 (+ 110us) server_negotiation.cc:407] Connection header received
0925 05:09:21.781838 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.781839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.781848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.781893 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:21.782012 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.782450 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.783177 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.783353 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.783400 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.783964 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:21.783965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:21.783966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:21.783969 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:21.783980 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:21.783984 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:21.784043 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.784083 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.784091 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:09:21.785622 571 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:21.787032 571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:21.792929 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.788962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35099 (local address 127.0.0.1:35994)
0925 05:09:21.789155 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.789189 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.789250 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.789646 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.789647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.789649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.789694 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.789695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.790693 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.790694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.791893 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.791896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.792660 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.792661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.792728 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.792740 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.792809 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.792855 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":33,"thread_start_us":59,"threads_started":1}
I20260925 05:09:21.792938 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.789120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.14.33.2:35099)
0925 05:09:21.789475 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.789479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.789487 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:21.789518 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.789520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.789532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.789636 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.789775 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.790610 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.791971 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.792576 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.792660 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.792745 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.792823 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.792857 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":90,"threads_started":1}
I20260925 05:09:21.793490 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.789301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.14.33.3:38351)
0925 05:09:21.789443 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.789446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.789455 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:09:21.789486 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.789488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.789507 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.789569 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.789849 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.790893 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.792070 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.792681 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.792747 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.792832 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.793080 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.793136 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260925 05:09:21.793514 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.788983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:36887 (local address 127.0.0.1:60736)
0925 05:09:21.789733 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.789743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.789801 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.790059 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.790059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.790061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.790107 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.790108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.791238 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.791239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.792215 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.792218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.793294 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.793295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.793337 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.793339 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.793395 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.793435 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":77,"threads_started":1}
I20260925 05:09:21.793537 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.789217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.14.33.1:36887)
0925 05:09:21.789416 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:21.789422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:21.789864 (+ 442us) server_negotiation.cc:407] Connection header received
0925 05:09:21.789909 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:21.789912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:21.789926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:21.789994 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:21.790186 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.791161 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.792307 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.793206 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.793297 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.793382 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:21.793416 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 05:09:21.793452 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260925 05:09:21.793854 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:21.788970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:38351 (local address 127.0.0.1:42388)
0925 05:09:21.789195 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:21.789205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:21.789257 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:21.789603 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:21.789606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:21.789609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:21.789666 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.789667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.790974 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.790974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.791985 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:21.791987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:21.792869 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:21.792870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:21.792931 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:21.792935 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:21.793367 (+ 432us) client_negotiation.cc:240] Negotiation successful
0925 05:09:21.793420 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260925 05:09:21.793768 826 tablet_service.cc:1467] Processing CreateTablet for tablet 551101a9e360443e93adb86ffad2f59f (DEFAULT_TABLE table=test-table [id=90b02489fa024191953e0096c6742998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:21.794119 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551101a9e360443e93adb86ffad2f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.794481 959 tablet_service.cc:1467] Processing CreateTablet for tablet 551101a9e360443e93adb86ffad2f59f (DEFAULT_TABLE table=test-table [id=90b02489fa024191953e0096c6742998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:21.794772 959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551101a9e360443e93adb86ffad2f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.796727 1039 tablet_bootstrap.cc:492] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Bootstrap starting.
I20260925 05:09:21.797356 1041 tablet_bootstrap.cc:492] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Bootstrap starting.
I20260925 05:09:21.797356 694 tablet_service.cc:1467] Processing CreateTablet for tablet 551101a9e360443e93adb86ffad2f59f (DEFAULT_TABLE table=test-table [id=90b02489fa024191953e0096c6742998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:21.797514 1039 tablet_bootstrap.cc:654] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.797690 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551101a9e360443e93adb86ffad2f59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:21.797804 1039 log.cc:824] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.798048 1041 tablet_bootstrap.cc:654] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.798353 1041 log.cc:824] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.798470 1039 tablet_bootstrap.cc:492] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: No bootstrap required, opened a new log
I20260925 05:09:21.798513 1039 ts_tablet_manager.cc:1397] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 05:09:21.799110 1041 tablet_bootstrap.cc:492] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: No bootstrap required, opened a new log
I20260925 05:09:21.799167 1041 ts_tablet_manager.cc:1397] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:21.800406 1039 raft_consensus.cc:348] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:21.800521 1039 raft_consensus.cc:374] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.800542 1039 raft_consensus.cc:729] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1ab145149647319528d128fb809b66, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.800653 1039 consensus_queue.cc:260] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:21.800915 894 heartbeater.cc:503] Master 127.14.33.62:38129 was elected leader, sending a full tablet report...
I20260925 05:09:21.801123 1039 ts_tablet_manager.cc:1428] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:09:21.801383 1045 tablet_bootstrap.cc:492] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: Bootstrap starting.
I20260925 05:09:21.801792 1041 raft_consensus.cc:348] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:21.801952 1041 raft_consensus.cc:374] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.802026 1045 tablet_bootstrap.cc:654] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:21.802023 1041 raft_consensus.cc:729] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52422d52240244b68ec5c2f5d8b5e185, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.802134 1041 consensus_queue.cc:260] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:21.802398 1041 ts_tablet_manager.cc:1428] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 05:09:21.802426 1045 log.cc:824] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:21.802456 1025 heartbeater.cc:503] Master 127.14.33.62:38129 was elected leader, sending a full tablet report...
I20260925 05:09:21.803120 1045 tablet_bootstrap.cc:492] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: No bootstrap required, opened a new log
I20260925 05:09:21.803202 1045 ts_tablet_manager.cc:1397] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:21.804865 1045 raft_consensus.cc:348] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:21.805004 1045 raft_consensus.cc:374] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:21.805049 1045 raft_consensus.cc:729] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954d55b299984a26b237cc2e6130ee00, State: Initialized, Role: FOLLOWER
I20260925 05:09:21.805202 1045 consensus_queue.cc:260] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:21.805434 1045 ts_tablet_manager.cc:1428] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:09:21.805501 760 heartbeater.cc:503] Master 127.14.33.62:38129 was elected leader, sending a full tablet report...
W20260925 05:09:21.899060 895 tablet.cc:2369] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:09:22.007467 761 tablet.cc:2369] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 05:09:22.020206 1026 tablet.cc:2369] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:22.112166 1046 raft_consensus.cc:482] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:22.112291 1046 raft_consensus.cc:504] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:22.112605 1046 leader_election.cc:288] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1ab145149647319528d128fb809b66 (127.14.33.2:35099), 954d55b299984a26b237cc2e6130ee00 (127.14.33.1:36887)
I20260925 05:09:22.115533 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:22.112893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:56285 (local address 127.14.33.2:35099)
0925 05:09:22.112938 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:22.112940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:22.113086 (+ 146us) server_negotiation.cc:407] Connection header received
0925 05:09:22.113158 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:22.113159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:22.113169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:22.113228 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:22.113410 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.113892 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.114825 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.115180 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.115231 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:22.115240 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:22.115420 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 05:09:22.115463 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:22.115536 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:22.112807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:35099 (local address 127.14.33.3:56285)
0925 05:09:22.113011 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:22.113021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:22.113078 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:22.113257 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:22.113258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:22.113260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:22.113332 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:22.113333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.113984 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.113984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:22.114717 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:22.114719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.115264 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.115265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:22.115341 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:22.115346 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:09:22.115402 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:09:22.115431 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260925 05:09:22.115602 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:22.112892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:52381 (local address 127.14.33.1:36887)
0925 05:09:22.112938 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:22.112940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:22.112952 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:09:22.112988 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:22.112989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:22.113000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:22.113080 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:22.113277 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.113815 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.114894 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.115257 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.115325 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:22.115335 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:22.115490 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 05:09:22.115525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:09:22.115638 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:22.112807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:36887 (local address 127.14.33.3:52381)
0925 05:09:22.112869 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:22.112882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:22.112930 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:22.113103 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:22.113104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:22.113106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:22.113200 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:22.113202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.113895 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.113896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:22.114717 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:22.114720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.115338 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.115339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:22.115410 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:22.115414 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:22.115468 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 05:09:22.115505 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":49}
I20260925 05:09:22.115916 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551101a9e360443e93adb86ffad2f59f" candidate_uuid: "52422d52240244b68ec5c2f5d8b5e185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954d55b299984a26b237cc2e6130ee00" is_pre_election: true
I20260925 05:09:22.115921 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551101a9e360443e93adb86ffad2f59f" candidate_uuid: "52422d52240244b68ec5c2f5d8b5e185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1ab145149647319528d128fb809b66" is_pre_election: true
I20260925 05:09:22.116076 714 raft_consensus.cc:2454] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52422d52240244b68ec5c2f5d8b5e185 in term 0.
I20260925 05:09:22.116076 846 raft_consensus.cc:2454] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52422d52240244b68ec5c2f5d8b5e185 in term 0.
I20260925 05:09:22.116325 914 leader_election.cc:302] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52422d52240244b68ec5c2f5d8b5e185, 954d55b299984a26b237cc2e6130ee00; no voters:
I20260925 05:09:22.116504 1046 raft_consensus.cc:2781] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:22.116575 1046 raft_consensus.cc:482] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:22.116592 1046 raft_consensus.cc:3037] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:22.117404 1046 raft_consensus.cc:504] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:22.117547 1046 leader_election.cc:288] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [CANDIDATE]: Term 1 election: Requested vote from peers fb1ab145149647319528d128fb809b66 (127.14.33.2:35099), 954d55b299984a26b237cc2e6130ee00 (127.14.33.1:36887)
I20260925 05:09:22.117720 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551101a9e360443e93adb86ffad2f59f" candidate_uuid: "52422d52240244b68ec5c2f5d8b5e185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954d55b299984a26b237cc2e6130ee00"
I20260925 05:09:22.117736 846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551101a9e360443e93adb86ffad2f59f" candidate_uuid: "52422d52240244b68ec5c2f5d8b5e185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1ab145149647319528d128fb809b66"
I20260925 05:09:22.117808 714 raft_consensus.cc:3037] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:22.117838 846 raft_consensus.cc:3037] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:22.118713 846 raft_consensus.cc:2454] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52422d52240244b68ec5c2f5d8b5e185 in term 1.
I20260925 05:09:22.118713 714 raft_consensus.cc:2454] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52422d52240244b68ec5c2f5d8b5e185 in term 1.
I20260925 05:09:22.118906 914 leader_election.cc:302] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52422d52240244b68ec5c2f5d8b5e185, 954d55b299984a26b237cc2e6130ee00; no voters:
I20260925 05:09:22.119004 1046 raft_consensus.cc:2781] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:22.119161 1046 raft_consensus.cc:686] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 1 LEADER]: Becoming Leader. State: Replica: 52422d52240244b68ec5c2f5d8b5e185, State: Running, Role: LEADER
I20260925 05:09:22.119258 1046 consensus_queue.cc:237] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } }
I20260925 05:09:22.120088 571 catalog_manager.cc:5495] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52422d52240244b68ec5c2f5d8b5e185 (127.14.33.3). New cstate: current_term: 1 leader_uuid: "52422d52240244b68ec5c2f5d8b5e185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52422d52240244b68ec5c2f5d8b5e185" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 38351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 } health_report { overall_health: UNKNOWN } } }
I20260925 05:09:22.151860 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:22.149619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.14.33.3:38351)
0925 05:09:22.149675 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:22.149677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:22.149685 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:22.149732 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:22.149733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:22.149750 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:22.149809 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:22.149933 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.150380 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.151056 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:22.151223 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:22.151271 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:22.151495 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:22.151581 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:22.151655 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:22.151723 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:09:22.151731 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:09:22.153820 846 raft_consensus.cc:1261] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 1 FOLLOWER]: Refusing update from remote peer 52422d52240244b68ec5c2f5d8b5e185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:22.153820 714 raft_consensus.cc:1261] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 1 FOLLOWER]: Refusing update from remote peer 52422d52240244b68ec5c2f5d8b5e185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:22.154177 1050 consensus_queue.cc:1035] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1ab145149647319528d128fb809b66" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:22.154328 1046 consensus_queue.cc:1035] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954d55b299984a26b237cc2e6130ee00" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:22.157337 1055 mvcc.cc:205] Tried to move back new op lower bound from 7333121892978962432 to 7333121892841181184. Current Snapshot: MvccSnapshot[applied={T|T < 7333121892978962432}]
I20260925 05:09:22.157554 1054 mvcc.cc:205] Tried to move back new op lower bound from 7333121892978962432 to 7333121892841181184. Current Snapshot: MvccSnapshot[applied={T|T < 7333121892978962432}]
I20260925 05:09:22.158824 572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:22.158916 572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:22.159667 572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73d2455e4b9c4b3ba8f3b92c5de2e060: Sending DeleteTablet for 3 replicas of tablet 551101a9e360443e93adb86ffad2f59f
I20260925 05:09:22.159999 959 tablet_service.cc:1514] Processing DeleteTablet for tablet 551101a9e360443e93adb86ffad2f59f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:22 UTC) from {username='slave'} at 127.0.0.1:42388
I20260925 05:09:22.160028 694 tablet_service.cc:1514] Processing DeleteTablet for tablet 551101a9e360443e93adb86ffad2f59f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:22 UTC) from {username='slave'} at 127.0.0.1:60736
I20260925 05:09:22.160240 1062 tablet_replica.cc:330] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: stopping tablet replica
I20260925 05:09:22.160327 826 tablet_service.cc:1514] Processing DeleteTablet for tablet 551101a9e360443e93adb86ffad2f59f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:22 UTC) from {username='slave'} at 127.0.0.1:35994
I20260925 05:09:22.160384 1062 raft_consensus.cc:2229] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:09:22.160467 1062 raft_consensus.cc:2258] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:22.160529 1064 tablet_replica.cc:330] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: stopping tablet replica
I20260925 05:09:22.160660 1063 tablet_replica.cc:330] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00: stopping tablet replica
I20260925 05:09:22.160665 1064 raft_consensus.cc:2229] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:22.160760 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 632
I20260925 05:09:22.160785 1063 raft_consensus.cc:2229] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:22.160887 1063 raft_consensus.cc:2258] T 551101a9e360443e93adb86ffad2f59f P 954d55b299984a26b237cc2e6130ee00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:22.161369 1064 raft_consensus.cc:2258] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:22.162279 1064 ts_tablet_manager.cc:1905] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:22.164162 1064 ts_tablet_manager.cc:1918] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:22.164223 1064 log.cc:1196] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/551101a9e360443e93adb86ffad2f59f
I20260925 05:09:22.164460 1064 ts_tablet_manager.cc:1939] T 551101a9e360443e93adb86ffad2f59f P fb1ab145149647319528d128fb809b66: Deleting consensus metadata
W20260925 05:09:22.166743 558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:36887 (error 108)
I20260925 05:09:22.167196 1062 ts_tablet_manager.cc:1905] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:22.167275 556 catalog_manager.cc:4841] TS fb1ab145149647319528d128fb809b66 (127.14.33.2:35099): tablet 551101a9e360443e93adb86ffad2f59f (table test-table [id=90b02489fa024191953e0096c6742998]) successfully deleted
W20260925 05:09:22.167516 1033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:22.167096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:36887 (local address 127.0.0.1:60746)
0925 05:09:22.167144 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:22.167156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:22.167193 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:22.167471 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:36887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 05:09:22.167714 558 catalog_manager.cc:4584] TS 954d55b299984a26b237cc2e6130ee00 (127.14.33.1:36887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 551101a9e360443e93adb86ffad2f59f: Network error: Client connection negotiation failed: client connection to 127.14.33.1:36887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 05:09:22.167722 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 763
I20260925 05:09:22.169298 1062 ts_tablet_manager.cc:1918] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:22.169373 1062 log.cc:1196] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/551101a9e360443e93adb86ffad2f59f
I20260925 05:09:22.169685 1062 ts_tablet_manager.cc:1939] T 551101a9e360443e93adb86ffad2f59f P 52422d52240244b68ec5c2f5d8b5e185: Deleting consensus metadata
I20260925 05:09:22.170082 557 catalog_manager.cc:4841] TS 52422d52240244b68ec5c2f5d8b5e185 (127.14.33.3:38351): tablet 551101a9e360443e93adb86ffad2f59f (table test-table [id=90b02489fa024191953e0096c6742998]) successfully deleted
I20260925 05:09:22.173946 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 897
I20260925 05:09:22.178455 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 541
2026-09-25T05:09:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (961 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:22 dist-test-slave-4dmq krb5kdc[1068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:22 dist-test-slave-4dmq krb5kdc[1068](info): set up 2 sockets
Sep 25 05:09:22 dist-test-slave-4dmq krb5kdc[1068](info): commencing operation
krb5kdc: starting...
W20260925 05:09:23.198032 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:23.315701 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:33257
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:33257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:23.394775 1081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:23.395009 1081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:23.395043 1081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:23.396126 1081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:23.396200 1081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:23.396232 1081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:23.396263 1081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:23.396284 1081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:23.397751 1081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:33257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:23.398118 1081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:23.398360 1081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:23.403254 1086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:23.403311 1081 server_base.cc:1034] running on GCE node
W20260925 05:09:23.403249 1087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.403249 1089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:23.403822 1081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:23.404112 1081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:23.405278 1081 hybrid_clock.cc:648] HybridClock initialized: now 1790312963405271 us; error 49 us; skew 500 ppm
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:23.407660 1081 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:23.408088 1081 webserver.cc:460] Webserver started at http://127.14.33.62:42657/ using document root <none> and password file <none>
I20260925 05:09:23.408249 1081 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:23.408286 1081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:23.408375 1081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:23.409348 1081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "0da592c756af47e889edbd7302d13e51"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "111e741a0d9f3dcbdf737def28f292e9"
server_key_iv: "05a03304fc74642bbee4e31b87726f19"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.409658 1081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "0da592c756af47e889edbd7302d13e51"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "111e741a0d9f3dcbdf737def28f292e9"
server_key_iv: "05a03304fc74642bbee4e31b87726f19"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.410807 1081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:23.411458 1096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.411748 1081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:23.411842 1081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "0da592c756af47e889edbd7302d13e51"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "111e741a0d9f3dcbdf737def28f292e9"
server_key_iv: "05a03304fc74642bbee4e31b87726f19"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.411911 1081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:23.432978 1081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:23.435269 1081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:23.435380 1081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:23.439083 1081 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:33257
I20260925 05:09:23.439136 1148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:33257 every 8 connection(s)
I20260925 05:09:23.439460 1081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:23.439877 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1081
I20260925 05:09:23.440054 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:23.440259 14468 external_mini_cluster.cc:1428] Setting key 3b345e3027b517e1f55957c502d8b8c3
I20260925 05:09:23.440626 1149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:23.443831 1149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51: Bootstrap starting.
I20260925 05:09:23.444486 1149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:23.444809 1149 log.cc:824] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:23.445547 1149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51: No bootstrap required, opened a new log
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:23.447363 1149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da592c756af47e889edbd7302d13e51" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33257 } }
I20260925 05:09:23.447522 1149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:23.447564 1149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da592c756af47e889edbd7302d13e51, State: Initialized, Role: FOLLOWER
I20260925 05:09:23.447688 1149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da592c756af47e889edbd7302d13e51" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33257 } }
I20260925 05:09:23.447777 1149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:23.447819 1149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:23.447875 1149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:23.448556 1149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da592c756af47e889edbd7302d13e51" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33257 } }
I20260925 05:09:23.448711 1149 leader_election.cc:302] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0da592c756af47e889edbd7302d13e51; no voters:
I20260925 05:09:23.448887 1149 leader_election.cc:288] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:23.448925 1154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:23.449090 1154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [term 1 LEADER]: Becoming Leader. State: Replica: 0da592c756af47e889edbd7302d13e51, State: Running, Role: LEADER
I20260925 05:09:23.449129 1149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:23.449285 1154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da592c756af47e889edbd7302d13e51" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33257 } }
I20260925 05:09:23.449705 1156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da592c756af47e889edbd7302d13e51. Latest consensus state: current_term: 1 leader_uuid: "0da592c756af47e889edbd7302d13e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da592c756af47e889edbd7302d13e51" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33257 } } }
I20260925 05:09:23.449832 1156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:23.450043 1163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:23.450587 1163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:23.449699 1155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0da592c756af47e889edbd7302d13e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0da592c756af47e889edbd7302d13e51" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 33257 } } }
I20260925 05:09:23.451329 1155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:23.451496 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.440750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.14.33.62:33257)
0925 05:09:23.440937 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.440945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.440980 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:09:23.441417 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.441424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.441474 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.441582 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:23.442367 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.443262 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.444364 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.444848 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.444964 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.447267 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:23.447271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:23.447271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:23.447282 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:23.449226 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.449661 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.449664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.449665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.449767 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.450208 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.450209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.450209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.450525 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:23.450666 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.451123 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.451255 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260925 05:09:23.454097 1163 catalog_manager.cc:1347] Generated new cluster ID: bd8ee1aca2ba492e8a8f2c4a6659092a
I20260925 05:09:23.454160 1163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:23.465309 1163 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:23.465983 1163 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:23.474020 1163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51: Generated new TSK 0
I20260925 05:09:23.474205 1163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.33.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
Entry for principal kudu/127.14.33.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab.
I20260925 05:09:23.508424 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33257
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:23.587865 1175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:23.588053 1175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:23.588074 1175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:23.589193 1175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:23.589228 1175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:23.589253 1175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:23.590593 1175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.1.keytab
--principal=kudu/127.14.33.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:23.590754 1175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:23.590909 1175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:23.595796 1183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.595893 1181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.595893 1180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:23.596108 1175 server_base.cc:1034] running on GCE node
I20260925 05:09:23.596276 1175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:23.596457 1175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:23.597605 1175 hybrid_clock.cc:648] HybridClock initialized: now 1790312963597589 us; error 31 us; skew 500 ppm
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:23.600104 1175 init.cc:377] Logged in from keytab as kudu/127.14.33.1@KRBTEST.COM (short username kudu)
I20260925 05:09:23.600526 1175 webserver.cc:460] Webserver started at http://127.14.33.1:40973/ using document root <none> and password file <none>
I20260925 05:09:23.600754 1175 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:23.600798 1175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:23.600939 1175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:23.602025 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "42af70eaacea485784cdefcc9845055a"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "38675d421caf445d81b9a758341265b0"
server_key_iv: "7f45ce3663fc6720bc1792fdbd111c68"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.602397 1175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "42af70eaacea485784cdefcc9845055a"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "38675d421caf445d81b9a758341265b0"
server_key_iv: "7f45ce3663fc6720bc1792fdbd111c68"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.603628 1175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:23.604281 1190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.604506 1175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:23.604598 1175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "42af70eaacea485784cdefcc9845055a"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "38675d421caf445d81b9a758341265b0"
server_key_iv: "7f45ce3663fc6720bc1792fdbd111c68"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.604664 1175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:23.623927 1175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:23.625895 1175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:23.626058 1175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:23.626276 1175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:23.626572 1175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:23.626605 1175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.626626 1175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:23.626672 1175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.632122 1175 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:43823
I20260925 05:09:23.632220 1303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:43823 every 8 connection(s)
I20260925 05:09:23.632494 1175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:23.632792 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1175
I20260925 05:09:23.632938 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:23.633190 14468 external_mini_cluster.cc:1428] Setting key 124d776836856e77ab938d721e384f9a
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.1@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.2@KRBTEST.COM; defaulting to no policy
I20260925 05:09:23.642493 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.633383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:56231 (local address 127.14.33.62:33257)
0925 05:09:23.633430 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.633432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.633568 (+ 136us) server_negotiation.cc:407] Connection header received
0925 05:09:23.634165 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.634165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.634175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.634219 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:23.635495 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.636006 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.637014 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.637317 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.637405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.639642 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:23.639644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:23.639645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:23.639653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:23.641231 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.641667 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.641669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.641670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.641734 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.641988 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.641989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.641989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.642091 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:23.642146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.642365 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.642431 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:09:23.642678 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.633296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33257 (local address 127.14.33.1:56231)
0925 05:09:23.633457 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:23.633494 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:23.634060 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:23.634290 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:23.634295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:23.635062 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:23.635371 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.635378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.636107 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.636108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.636903 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.636905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.637416 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.637417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.637544 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.638020 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:23.638027 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:23.639541 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:23.641334 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:23.641340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:23.641342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:23.641557 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:23.641805 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:23.641806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:23.641806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:23.641914 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:23.642151 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:23.642155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:23.642255 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 05:09:23.642333 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 05:09:23.642443 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260925 05:09:23.643108 1304 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33257
I20260925 05:09:23.643215 1304 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:23.643385 1304 heartbeater.cc:511] Master 127.14.33.62:33257 requested a full tablet report, sending...
I20260925 05:09:23.643929 1113 ts_manager.cc:194] Registered new tserver with Master: 42af70eaacea485784cdefcc9845055a (127.14.33.1:43823)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.2@KRBTEST.COM" created.
I20260925 05:09:23.644726 1113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.1@KRBTEST.COM'} at 127.14.33.1:56231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
Entry for principal kudu/127.14.33.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab.
I20260925 05:09:23.660633 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33257
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:23.738934 1309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:23.739161 1309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:23.739193 1309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:23.740289 1309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:23.740360 1309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:23.740403 1309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:23.741918 1309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.2.keytab
--principal=kudu/127.14.33.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:23.742177 1309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:23.742404 1309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:23.747166 1314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.747195 1317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.747170 1315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:23.747460 1309 server_base.cc:1034] running on GCE node
I20260925 05:09:23.747701 1309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:23.747948 1309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:23.749089 1309 hybrid_clock.cc:648] HybridClock initialized: now 1790312963749077 us; error 27 us; skew 500 ppm
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:23.751725 1309 init.cc:377] Logged in from keytab as kudu/127.14.33.2@KRBTEST.COM (short username kudu)
I20260925 05:09:23.752174 1309 webserver.cc:460] Webserver started at http://127.14.33.2:35763/ using document root <none> and password file <none>
I20260925 05:09:23.752404 1309 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:23.752473 1309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:23.752624 1309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:23.753635 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "6b1e621569da4a58b7d510e55d90648d"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "f4ad0cf13154a74c24f2e5c3b904755a"
server_key_iv: "eb1786402e98ce7427efbce2241b2b40"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.753985 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "6b1e621569da4a58b7d510e55d90648d"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "f4ad0cf13154a74c24f2e5c3b904755a"
server_key_iv: "eb1786402e98ce7427efbce2241b2b40"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.755213 1309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:23.755913 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.756078 1309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:23.756188 1309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "6b1e621569da4a58b7d510e55d90648d"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "f4ad0cf13154a74c24f2e5c3b904755a"
server_key_iv: "eb1786402e98ce7427efbce2241b2b40"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.756269 1309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:23.768931 1309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:23.770653 1309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:23.770816 1309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:23.771034 1309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:23.771322 1309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:23.771382 1309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.771445 1309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:23.771472 1309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.777277 1309 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:37013
I20260925 05:09:23.777359 1437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:37013 every 8 connection(s)
I20260925 05:09:23.777695 1309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.2@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:23.784782 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1309
I20260925 05:09:23.784933 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:23.785157 14468 external_mini_cluster.cc:1428] Setting key de8726db1b7e8d660ed8cfe9932e5f70
I20260925 05:09:23.786289 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.778387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:58477 (local address 127.14.33.62:33257)
0925 05:09:23.778462 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.778464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.778647 (+ 183us) server_negotiation.cc:407] Connection header received
0925 05:09:23.779069 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.779070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.779080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.779126 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:23.779939 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.780425 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.781465 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.781652 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.781700 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.783571 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:23.783573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:23.783573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:23.783580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:23.784779 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.785311 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.785312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.785313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.785391 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.785662 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.785663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.785663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.785792 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:23.785858 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.786176 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.786233 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 05:09:23.786553 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.778341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33257 (local address 127.14.33.2:58477)
0925 05:09:23.778531 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:23.778564 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:23.778990 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:23.779179 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:23.779182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:23.779690 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:23.779868 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.779872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.780513 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.780513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.781390 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.781393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.781724 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.781724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.781810 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.782209 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:23.782216 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:23.783455 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:23.784877 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:23.784886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:23.784888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:23.785204 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:23.785469 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:23.785470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:23.785470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:23.785593 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:23.785861 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:23.785900 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:23.786043 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 05:09:23.786153 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 05:09:23.786297 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260925 05:09:23.787098 1438 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33257
I20260925 05:09:23.787189 1438 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:23.787386 1438 heartbeater.cc:511] Master 127.14.33.62:33257 requested a full tablet report, sending...
I20260925 05:09:23.787796 1113 ts_manager.cc:194] Registered new tserver with Master: 6b1e621569da4a58b7d510e55d90648d (127.14.33.2:37013)
I20260925 05:09:23.788177 1113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.2@KRBTEST.COM'} at 127.14.33.2:58477
WARNING: no policy specified for kudu/127.14.33.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
Entry for principal kudu/127.14.33.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab.
I20260925 05:09:23.811955 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33257
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:23.890491 1443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:23.890687 1443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:23.890710 1443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:23.891826 1443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:23.891856 1443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:23.891881 1443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:23.893246 1443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.3.keytab
--principal=kudu/127.14.33.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:23.893428 1443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:23.893584 1443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:09:23.898696 1443 server_base.cc:1034] running on GCE node
W20260925 05:09:23.898648 1448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.898648 1451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:23.898626 1449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:23.899029 1443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:23.899224 1443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:23.900365 1443 hybrid_clock.cc:648] HybridClock initialized: now 1790312963900354 us; error 27 us; skew 500 ppm
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:23.902559 1443 init.cc:377] Logged in from keytab as kudu/127.14.33.3@KRBTEST.COM (short username kudu)
I20260925 05:09:23.902928 1443 webserver.cc:460] Webserver started at http://127.14.33.3:37231/ using document root <none> and password file <none>
I20260925 05:09:23.903108 1443 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:23.903147 1443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:23.903226 1443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:23.904201 1443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "5ff196d8c2d04d458b136b446914e49f"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "194866668e57a5aec69404bc2c28b013"
server_key_iv: "8cbee28a6aa6c51140413eec5ac99266"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.904506 1443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "5ff196d8c2d04d458b136b446914e49f"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "194866668e57a5aec69404bc2c28b013"
server_key_iv: "8cbee28a6aa6c51140413eec5ac99266"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.905853 1443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:23.906527 1458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.906740 1443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:23.906808 1443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "5ff196d8c2d04d458b136b446914e49f"
format_stamp: "Formatted at 2026-09-25 05:09:23 on dist-test-slave-4dmq"
server_key: "194866668e57a5aec69404bc2c28b013"
server_key_iv: "8cbee28a6aa6c51140413eec5ac99266"
server_key_version: "encryptionkey@0"
I20260925 05:09:23.906898 1443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:23.920507 1443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:23.922438 1443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:23.922605 1443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:23.922837 1443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:23.923141 1443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:23.923173 1443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.923195 1443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:23.923246 1443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:23.929203 1443 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:42941
I20260925 05:09:23.929256 1572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:42941 every 8 connection(s)
I20260925 05:09:23.929553 1443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
Sep 25 05:09:23 dist-test-slave-4dmq krb5kdc[1068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.3@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:23.936597 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1443
I20260925 05:09:23.936718 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:23.936911 14468 external_mini_cluster.cc:1428] Setting key 33624c4ca47d8f84ecbe2e9606029a39
I20260925 05:09:23.938614 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.930245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:57451 (local address 127.14.33.62:33257)
0925 05:09:23.930323 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.930326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.930561 (+ 235us) server_negotiation.cc:407] Connection header received
0925 05:09:23.931005 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.931006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.931017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.931065 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:23.931928 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.932419 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.933401 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.933602 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.933692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.935577 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:23.935579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:23.935579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:23.935585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:23.937353 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.937822 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.937823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.937824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.937887 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.938122 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.938122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.938123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.938222 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:23.938282 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.938484 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.938542 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:09:23.938786 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.930214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:33257 (local address 127.14.33.3:57451)
0925 05:09:23.930424 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:23.930465 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:23.930917 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:23.931125 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:23.931128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:23.931667 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:23.931847 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.931851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.932502 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.932502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.933287 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.933289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.933703 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.933704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.933831 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.934216 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:23.934223 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:23.935469 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:23.937430 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:23.937438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:23.937440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:23.937737 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:23.937957 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:23.937958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:23.937958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:23.938059 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:23.938287 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:23.938291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:23.938395 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 05:09:23.938474 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 05:09:23.938594 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
I20260925 05:09:23.939247 1574 heartbeater.cc:347] Connected to a master server at 127.14.33.62:33257
I20260925 05:09:23.939349 1574 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:23.939527 1574 heartbeater.cc:511] Master 127.14.33.62:33257 requested a full tablet report, sending...
I20260925 05:09:23.939874 1113 ts_manager.cc:194] Registered new tserver with Master: 5ff196d8c2d04d458b136b446914e49f (127.14.33.3:42941)
I20260925 05:09:23.940261 1113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.33.3@KRBTEST.COM'} at 127.14.33.3:57451
I20260925 05:09:23.948563 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:23.955667 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.950045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.14.33.62:33257)
0925 05:09:23.950107 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.950113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.950148 (+ 35us) server_negotiation.cc:407] Connection header received
0925 05:09:23.950196 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.950196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.950204 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.950305 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:23.950846 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.951425 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.952105 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.952291 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.952341 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.953442 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:23.953443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:23.953443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:23.953449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:23.954638 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.954983 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.954984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.954984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.955043 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:23.955240 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:23.955240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:23.955241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:23.955368 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:23.955444 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.955551 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.955608 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 05:09:23.957113 1113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:23.958760 1113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:23.965036 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.961190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.14.33.3:42941)
0925 05:09:23.961415 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.961418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.961429 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:09:23.961468 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.961470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.961485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.961560 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:23.961797 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.962501 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.963911 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.964612 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964685 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.964740 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.964905 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.964957 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260925 05:09:23.965029 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.960886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:42941 (local address 127.0.0.1:54340)
0925 05:09:23.961200 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:23.961212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:23.961276 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:23.961644 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:23.961648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:23.961652 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:23.961717 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.961719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.962623 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.962624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.963822 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.963825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.964707 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.964786 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.964797 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 05:09:23.964880 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:09:23.964944 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":92,"threads_started":1}
I20260925 05:09:23.965246 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.961187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.14.33.2:37013)
0925 05:09:23.961335 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.961340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.961393 (+ 53us) server_negotiation.cc:407] Connection header received
0925 05:09:23.961459 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.961461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.961476 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.961547 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:23.961806 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.962919 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964176 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.964677 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964746 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.964801 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.965067 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.965115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260925 05:09:23.965314 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.960885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:43823 (local address 127.0.0.1:51014)
0925 05:09:23.961029 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:23.961064 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:23.961122 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:23.961767 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:23.961768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:23.961769 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:23.961811 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.961812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.963085 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.963086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.964089 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.964091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.964995 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.965060 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.965065 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:09:23.965096 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 05:09:23.965161 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260925 05:09:23.965031 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.960916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:37013 (local address 127.0.0.1:43528)
0925 05:09:23.961342 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:23.961350 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:23.961391 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:23.961645 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:23.961647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:23.961649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:23.961712 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.961714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.963029 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.963030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.964090 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:23.964092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.964833 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:23.964893 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.964896 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:23.964941 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 05:09:23.964969 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":29,"thread_start_us":53,"threads_started":1}
I20260925 05:09:23.965487 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:23.961396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.14.33.1:43823)
0925 05:09:23.961600 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:23.961605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:23.961613 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:23.961651 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:23.961653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:23.961667 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:23.961902 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:23.961972 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.963004 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.964180 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:23.964902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:23.965279 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:23.965343 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:23.965368 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 05:09:23.965409 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260925 05:09:23.966221 1238 tablet_service.cc:1467] Processing CreateTablet for tablet b707db7de2464f17bfeb861e45d38a1d (DEFAULT_TABLE table=test-table [id=90414dc0713e4057914ec8b3f378f720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:23.966316 1372 tablet_service.cc:1467] Processing CreateTablet for tablet b707db7de2464f17bfeb861e45d38a1d (DEFAULT_TABLE table=test-table [id=90414dc0713e4057914ec8b3f378f720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:23.966413 1507 tablet_service.cc:1467] Processing CreateTablet for tablet b707db7de2464f17bfeb861e45d38a1d (DEFAULT_TABLE table=test-table [id=90414dc0713e4057914ec8b3f378f720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:23.966599 1238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b707db7de2464f17bfeb861e45d38a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:23.966668 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b707db7de2464f17bfeb861e45d38a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:23.966665 1372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b707db7de2464f17bfeb861e45d38a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:23.969426 1588 tablet_bootstrap.cc:492] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: Bootstrap starting.
I20260925 05:09:23.969871 1590 tablet_bootstrap.cc:492] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Bootstrap starting.
I20260925 05:09:23.970340 1588 tablet_bootstrap.cc:654] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:23.970513 1590 tablet_bootstrap.cc:654] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:23.970978 1588 log.cc:824] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:23.971063 1590 log.cc:824] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:23.971374 1589 tablet_bootstrap.cc:492] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Bootstrap starting.
I20260925 05:09:23.971752 1588 tablet_bootstrap.cc:492] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: No bootstrap required, opened a new log
I20260925 05:09:23.971783 1590 tablet_bootstrap.cc:492] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: No bootstrap required, opened a new log
I20260925 05:09:23.971841 1590 ts_tablet_manager.cc:1397] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:23.971843 1588 ts_tablet_manager.cc:1397] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 05:09:23.972010 1589 tablet_bootstrap.cc:654] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:23.972546 1589 log.cc:824] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:23.973345 1589 tablet_bootstrap.cc:492] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: No bootstrap required, opened a new log
I20260925 05:09:23.973414 1589 ts_tablet_manager.cc:1397] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:23.973826 1588 raft_consensus.cc:348] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:23.974004 1588 raft_consensus.cc:374] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:23.974048 1588 raft_consensus.cc:729] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42af70eaacea485784cdefcc9845055a, State: Initialized, Role: FOLLOWER
I20260925 05:09:23.974047 1590 raft_consensus.cc:348] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:23.974159 1588 consensus_queue.cc:260] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:23.974220 1590 raft_consensus.cc:374] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:23.974256 1590 raft_consensus.cc:729] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e621569da4a58b7d510e55d90648d, State: Initialized, Role: FOLLOWER
I20260925 05:09:23.974396 1588 ts_tablet_manager.cc:1428] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:09:23.974514 1304 heartbeater.cc:503] Master 127.14.33.62:33257 was elected leader, sending a full tablet report...
I20260925 05:09:23.974432 1590 consensus_queue.cc:260] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:23.974977 1590 ts_tablet_manager.cc:1428] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 05:09:23.975097 1438 heartbeater.cc:503] Master 127.14.33.62:33257 was elected leader, sending a full tablet report...
I20260925 05:09:23.975435 1589 raft_consensus.cc:348] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:23.975610 1589 raft_consensus.cc:374] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:23.975656 1589 raft_consensus.cc:729] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff196d8c2d04d458b136b446914e49f, State: Initialized, Role: FOLLOWER
I20260925 05:09:23.975754 1589 consensus_queue.cc:260] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:23.976028 1589 ts_tablet_manager.cc:1428] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 05:09:23.976058 1574 heartbeater.cc:503] Master 127.14.33.62:33257 was elected leader, sending a full tablet report...
I20260925 05:09:23.999534 1595 raft_consensus.cc:482] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:23.999670 1595 raft_consensus.cc:504] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:24.000069 1595 leader_election.cc:288] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff196d8c2d04d458b136b446914e49f (127.14.33.3:42941), 6b1e621569da4a58b7d510e55d90648d (127.14.33.2:37013)
I20260925 05:09:24.002971 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.000312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:48367 (local address 127.14.33.3:42941)
0925 05:09:24.000364 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.000366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.000527 (+ 161us) server_negotiation.cc:407] Connection header received
0925 05:09:24.000552 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.000553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.000564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.000616 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.000804 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.001365 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.002259 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.002625 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.002679 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.002687 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.002864 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.002909 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 05:09:24.002982 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.000228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:42941 (local address 127.14.33.1:48367)
0925 05:09:24.000422 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.000431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.000489 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.000650 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.000651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.000666 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.000724 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.000726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.001442 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.001443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.002158 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.002160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.002710 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.002711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.002781 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.002785 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.002836 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.002870 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":11,"thread_start_us":103,"threads_started":1}
I20260925 05:09:24.003042 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.000307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:55229 (local address 127.14.33.2:37013)
0925 05:09:24.000346 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.000349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.000371 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:09:24.000397 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.000398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.000408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.000484 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.000673 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.001292 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.002265 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.002638 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.002726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.002735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.002936 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.002972 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:09:24.003280 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.000228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:37013 (local address 127.14.33.1:55229)
0925 05:09:24.000277 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.000288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.000341 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.000520 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.000521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.000523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.000598 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.000600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.001367 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.001368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.002156 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.002158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.002755 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.002756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.002853 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.002857 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.002913 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.002950 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":25}
I20260925 05:09:24.003324 1527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b707db7de2464f17bfeb861e45d38a1d" candidate_uuid: "42af70eaacea485784cdefcc9845055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff196d8c2d04d458b136b446914e49f" is_pre_election: true
I20260925 05:09:24.003459 1527 raft_consensus.cc:2454] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42af70eaacea485784cdefcc9845055a in term 0.
I20260925 05:09:24.003676 1191 leader_election.cc:302] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42af70eaacea485784cdefcc9845055a, 5ff196d8c2d04d458b136b446914e49f; no voters:
I20260925 05:09:24.003654 1392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b707db7de2464f17bfeb861e45d38a1d" candidate_uuid: "42af70eaacea485784cdefcc9845055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e621569da4a58b7d510e55d90648d" is_pre_election: true
I20260925 05:09:24.003805 1392 raft_consensus.cc:2454] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42af70eaacea485784cdefcc9845055a in term 0.
I20260925 05:09:24.003799 1595 raft_consensus.cc:2781] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:24.003851 1595 raft_consensus.cc:482] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:24.003877 1595 raft_consensus.cc:3037] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.004554 1595 raft_consensus.cc:504] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:24.004704 1595 leader_election.cc:288] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [CANDIDATE]: Term 1 election: Requested vote from peers 5ff196d8c2d04d458b136b446914e49f (127.14.33.3:42941), 6b1e621569da4a58b7d510e55d90648d (127.14.33.2:37013)
I20260925 05:09:24.004851 1527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b707db7de2464f17bfeb861e45d38a1d" candidate_uuid: "42af70eaacea485784cdefcc9845055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff196d8c2d04d458b136b446914e49f"
I20260925 05:09:24.004880 1392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b707db7de2464f17bfeb861e45d38a1d" candidate_uuid: "42af70eaacea485784cdefcc9845055a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1e621569da4a58b7d510e55d90648d"
I20260925 05:09:24.004951 1392 raft_consensus.cc:3037] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.004936 1527 raft_consensus.cc:3037] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.005687 1392 raft_consensus.cc:2454] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42af70eaacea485784cdefcc9845055a in term 1.
I20260925 05:09:24.005764 1527 raft_consensus.cc:2454] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42af70eaacea485784cdefcc9845055a in term 1.
I20260925 05:09:24.005962 1193 leader_election.cc:302] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42af70eaacea485784cdefcc9845055a, 6b1e621569da4a58b7d510e55d90648d; no voters:
I20260925 05:09:24.006060 1595 raft_consensus.cc:2781] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:24.006194 1595 raft_consensus.cc:686] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 1 LEADER]: Becoming Leader. State: Replica: 42af70eaacea485784cdefcc9845055a, State: Running, Role: LEADER
I20260925 05:09:24.006289 1595 consensus_queue.cc:237] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } }
I20260925 05:09:24.007019 1113 catalog_manager.cc:5495] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a reported cstate change: term changed from 0 to 1, leader changed from <none> to 42af70eaacea485784cdefcc9845055a (127.14.33.1). New cstate: current_term: 1 leader_uuid: "42af70eaacea485784cdefcc9845055a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42af70eaacea485784cdefcc9845055a" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 43823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 } health_report { overall_health: UNKNOWN } } }
I20260925 05:09:24.022358 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.019808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.14.33.1:43823)
0925 05:09:24.019866 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.019868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.019878 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:09:24.019921 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.019922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.019939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.020021 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:24.020167 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.020651 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.021518 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.021726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.021821 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.022004 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:24.022104 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:24.022205 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.022230 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.022246 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:09:24.024010 1527 raft_consensus.cc:1261] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 1 FOLLOWER]: Refusing update from remote peer 42af70eaacea485784cdefcc9845055a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:24.024037 1392 raft_consensus.cc:1261] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 1 FOLLOWER]: Refusing update from remote peer 42af70eaacea485784cdefcc9845055a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:24.024358 1595 consensus_queue.cc:1035] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff196d8c2d04d458b136b446914e49f" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:24.024507 1599 consensus_queue.cc:1035] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1e621569da4a58b7d510e55d90648d" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:24.026813 1603 mvcc.cc:205] Tried to move back new op lower bound from 7333121900639895552 to 7333121900570288128. Current Snapshot: MvccSnapshot[applied={T|T < 7333121900639895552}]
I20260925 05:09:24.028813 1113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:24.028903 1113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:24.029706 1113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0da592c756af47e889edbd7302d13e51: Sending DeleteTablet for 3 replicas of tablet b707db7de2464f17bfeb861e45d38a1d
I20260925 05:09:24.030033 1238 tablet_service.cc:1514] Processing DeleteTablet for tablet b707db7de2464f17bfeb861e45d38a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:24 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:51014
I20260925 05:09:24.030114 1372 tablet_service.cc:1514] Processing DeleteTablet for tablet b707db7de2464f17bfeb861e45d38a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:24 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:43528
I20260925 05:09:24.030117 1507 tablet_service.cc:1514] Processing DeleteTablet for tablet b707db7de2464f17bfeb861e45d38a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:24 UTC) from {username='kudu', principal='kudu/127.14.33.62@KRBTEST.COM'} at 127.0.0.1:54340
I20260925 05:09:24.030318 1612 tablet_replica.cc:330] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: stopping tablet replica
I20260925 05:09:24.030339 1611 tablet_replica.cc:330] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a: stopping tablet replica
I20260925 05:09:24.030416 1612 raft_consensus.cc:2229] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:24.030515 1612 raft_consensus.cc:2258] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:24.030534 1611 raft_consensus.cc:2229] T b707db7de2464f17bfeb861e45d38a1d P 42af70eaacea485784cdefcc9845055a [term 1 LEADER]: Raft consensus shutting down.
I20260925 05:09:24.030610 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1175
I20260925 05:09:24.031543 1613 tablet_replica.cc:330] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: stopping tablet replica
I20260925 05:09:24.031653 1613 raft_consensus.cc:2229] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:24.031765 1613 raft_consensus.cc:2258] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:24.032104 1613 ts_tablet_manager.cc:1905] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:24.033546 1612 ts_tablet_manager.cc:1905] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:24.033752 1613 ts_tablet_manager.cc:1918] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:24.033818 1613 log.cc:1196] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/b707db7de2464f17bfeb861e45d38a1d
I20260925 05:09:24.034088 1613 ts_tablet_manager.cc:1939] T b707db7de2464f17bfeb861e45d38a1d P 5ff196d8c2d04d458b136b446914e49f: Deleting consensus metadata
I20260925 05:09:24.034504 1097 catalog_manager.cc:4841] TS 5ff196d8c2d04d458b136b446914e49f (127.14.33.3:42941): tablet b707db7de2464f17bfeb861e45d38a1d (table test-table [id=90414dc0713e4057914ec8b3f378f720]) successfully deleted
I20260925 05:09:24.034999 1612 ts_tablet_manager.cc:1918] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:24.035049 1612 log.cc:1196] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/b707db7de2464f17bfeb861e45d38a1d
I20260925 05:09:24.035280 1612 ts_tablet_manager.cc:1939] T b707db7de2464f17bfeb861e45d38a1d P 6b1e621569da4a58b7d510e55d90648d: Deleting consensus metadata
I20260925 05:09:24.035670 1099 catalog_manager.cc:4841] TS 6b1e621569da4a58b7d510e55d90648d (127.14.33.2:37013): tablet b707db7de2464f17bfeb861e45d38a1d (table test-table [id=90414dc0713e4057914ec8b3f378f720]) successfully deleted
W20260925 05:09:24.036674 1098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:43823 (error 108)
I20260925 05:09:24.036840 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1309
I20260925 05:09:24.041458 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.041306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:43823 (local address 127.0.0.1:51040)
0925 05:09:24.041371 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.041421 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:43823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:09:24.041502 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1443
W20260925 05:09:24.041641 1098 catalog_manager.cc:4584] TS 42af70eaacea485784cdefcc9845055a (127.14.33.1:43823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b707db7de2464f17bfeb861e45d38a1d: Network error: Client connection negotiation failed: client connection to 127.14.33.1:43823: connect: Connection refused (error 111)
I20260925 05:09:24.046334 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1081
2026-09-25T05:09:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1868 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T05:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:24Z Disabled control of system clock
I20260925 05:09:24.073432 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:45461
--webserver_interface=127.14.33.62
--webserver_port=0
--builtin_ntp_servers=127.14.33.20:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:45461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:24.152835 1618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:24.153043 1618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:24.153062 1618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:24.154217 1618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:24.154258 1618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:24.154271 1618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:24.154284 1618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:24.154294 1618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:24.155544 1618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:45461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:45461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:24.155720 1618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:24.155939 1618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:24.160585 1623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:24.160772 1618 server_base.cc:1034] running on GCE node
W20260925 05:09:24.160610 1626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:24.160578 1624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:24.161182 1618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:24.161474 1618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:24.162638 1618 hybrid_clock.cc:648] HybridClock initialized: now 1790312964162631 us; error 39 us; skew 500 ppm
I20260925 05:09:24.163779 1618 webserver.cc:460] Webserver started at http://127.14.33.62:32781/ using document root <none> and password file <none>
I20260925 05:09:24.164019 1618 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:24.164091 1618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:24.164222 1618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:24.165295 1618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "99bdb1bdb37cab0b0779dbb1f9dffa32"
server_key_iv: "39d4549b7c799a7fe6d428a12f7885ae"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.165669 1618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "99bdb1bdb37cab0b0779dbb1f9dffa32"
server_key_iv: "39d4549b7c799a7fe6d428a12f7885ae"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.166945 1618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:09:24.167730 1632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.167917 1618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:24.167981 1618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "99bdb1bdb37cab0b0779dbb1f9dffa32"
server_key_iv: "39d4549b7c799a7fe6d428a12f7885ae"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.168031 1618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:24.180552 1618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:24.180770 1618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:24.180866 1618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:24.184551 1618 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:45461
I20260925 05:09:24.184643 1684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:45461 every 8 connection(s)
I20260925 05:09:24.185019 1618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:24.186225 1685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:24.187299 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1618
I20260925 05:09:24.187402 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:24.187601 14468 external_mini_cluster.cc:1428] Setting key b3979b97995681212d53f19bd3f5d018
I20260925 05:09:24.189687 1685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b: Bootstrap starting.
I20260925 05:09:24.190363 1685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:24.190709 1685 log.cc:824] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:24.191466 1685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b: No bootstrap required, opened a new log
I20260925 05:09:24.193759 1685 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45461 } }
I20260925 05:09:24.193935 1685 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:24.194008 1685 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5b76dcaf3443ca955dbff3f5b25d4b, State: Initialized, Role: FOLLOWER
I20260925 05:09:24.194118 1685 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45461 } }
I20260925 05:09:24.194203 1685 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:24.194237 1685 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:24.194310 1685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.195045 1685 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45461 } }
I20260925 05:09:24.195158 1685 leader_election.cc:302] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5b76dcaf3443ca955dbff3f5b25d4b; no voters:
I20260925 05:09:24.195211 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.188162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.14.33.62:45461)
0925 05:09:24.188363 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.188371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.189314 (+ 943us) server_negotiation.cc:407] Connection header received
0925 05:09:24.189927 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.189935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.190030 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.190180 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:24.190459 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.191695 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.192732 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.193411 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.193546 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.194449 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:24.194453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:24.194453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:24.194647 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:24.194687 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:24.194700 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:24.194739 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.194896 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.194977 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260925 05:09:24.195325 1685 leader_election.cc:288] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:24.195390 1690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:24.195528 1690 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [term 1 LEADER]: Becoming Leader. State: Replica: ba5b76dcaf3443ca955dbff3f5b25d4b, State: Running, Role: LEADER
I20260925 05:09:24.195570 1685 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:24.195608 1690 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45461 } }
I20260925 05:09:24.196012 1691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45461 } } }
I20260925 05:09:24.196090 1691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:24.196254 1692 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5b76dcaf3443ca955dbff3f5b25d4b. Latest consensus state: current_term: 1 leader_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5b76dcaf3443ca955dbff3f5b25d4b" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 45461 } } }
I20260925 05:09:24.196317 1692 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:24.196297 1698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:24.196928 1698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:24.198565 1698 catalog_manager.cc:1347] Generated new cluster ID: a29f78daa0b844c68e885fca04aaafc8
I20260925 05:09:24.198632 1698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:24.207540 1698 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:24.208024 1698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:24.221909 1698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b: Generated new TSK 0
I20260925 05:09:24.222106 1698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:24.223556 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.1:0
--local_ip_for_outbound_sockets=127.14.33.1
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45461
--builtin_ntp_servers=127.14.33.20:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 05:09:24.309082 1709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:24.309351 1709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:24.309384 1709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:24.310531 1709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:24.310604 1709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:24.310648 1709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.1
I20260925 05:09:24.311998 1709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.33.1
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.1
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:24.312253 1709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:24.312548 1709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:24.318217 1715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:24.318255 1714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:24.318216 1717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:24.318568 1709 server_base.cc:1034] running on GCE node
I20260925 05:09:24.318763 1709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:24.319031 1709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:24.320190 1709 hybrid_clock.cc:648] HybridClock initialized: now 1790312964320171 us; error 36 us; skew 500 ppm
I20260925 05:09:24.321382 1709 webserver.cc:460] Webserver started at http://127.14.33.1:38973/ using document root <none> and password file <none>
I20260925 05:09:24.321633 1709 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:24.321678 1709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:24.321822 1709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:24.322827 1709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data/instance:
uuid: "bf5ec4e485d34f8983000c3cc864142c"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "ce29eb3f9b8f654e75532b3f6e4535a8"
server_key_iv: "62983d7addab0c4f11fd3c23650c2134"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.323199 1709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance:
uuid: "bf5ec4e485d34f8983000c3cc864142c"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "ce29eb3f9b8f654e75532b3f6e4535a8"
server_key_iv: "62983d7addab0c4f11fd3c23650c2134"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.324460 1709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:24.325243 1723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.325403 1709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:24.325475 1709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal
uuid: "bf5ec4e485d34f8983000c3cc864142c"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "ce29eb3f9b8f654e75532b3f6e4535a8"
server_key_iv: "62983d7addab0c4f11fd3c23650c2134"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.325525 1709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:24.340472 1709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:24.340699 1709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:24.340793 1709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:24.340956 1709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:24.341262 1709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:24.341301 1709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.341322 1709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:24.341334 1709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.346757 1709 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.1:37733
I20260925 05:09:24.346843 1836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.1:37733 every 8 connection(s)
I20260925 05:09:24.347142 1709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/data/info.pb
I20260925 05:09:24.351784 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.347836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:39497 (local address 127.14.33.62:45461)
0925 05:09:24.347886 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.347889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.348103 (+ 214us) server_negotiation.cc:407] Connection header received
0925 05:09:24.348538 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.348539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.348552 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.348602 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:24.349247 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.349727 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.350648 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.350833 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.350881 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.351562 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:24.351563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:24.351564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:24.351567 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:24.351579 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:24.351583 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:24.351624 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.351710 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.351728 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:24.351996 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.347824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45461 (local address 127.14.33.1:39497)
0925 05:09:24.347984 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.348014 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.348446 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.348656 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.348659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.348666 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:24.349167 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.349171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.349813 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.349814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.350576 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.350577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.350911 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.350912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.351013 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.351440 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:24.351441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:24.351467 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:24.351488 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:24.351641 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:24.351645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:24.351647 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.351697 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.351768 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260925 05:09:24.352069 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1709
I20260925 05:09:24.352195 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-0/wal/instance
I20260925 05:09:24.352411 14468 external_mini_cluster.cc:1428] Setting key e403c115b1a54f645f790115446f1f82
I20260925 05:09:24.352553 1837 heartbeater.cc:347] Connected to a master server at 127.14.33.62:45461
I20260925 05:09:24.352650 1837 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:24.352842 1837 heartbeater.cc:511] Master 127.14.33.62:45461 requested a full tablet report, sending...
I20260925 05:09:24.353356 1649 ts_manager.cc:194] Registered new tserver with Master: bf5ec4e485d34f8983000c3cc864142c (127.14.33.1:37733)
I20260925 05:09:24.353371 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.2:0
--local_ip_for_outbound_sockets=127.14.33.2
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45461
--builtin_ntp_servers=127.14.33.20:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:09:24.353983 1649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.1:39497
W20260925 05:09:24.432013 1840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:24.432212 1840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:24.432233 1840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:24.433413 1840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:24.433451 1840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:24.433475 1840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.2
I20260925 05:09:24.434805 1840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.33.2
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.2
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:24.435057 1840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:24.435387 1840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:24.440263 1846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:24.440245 1848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:24.440248 1845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:24.440548 1840 server_base.cc:1034] running on GCE node
I20260925 05:09:24.440797 1840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:24.441035 1840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:24.442193 1840 hybrid_clock.cc:648] HybridClock initialized: now 1790312964442188 us; error 38 us; skew 500 ppm
I20260925 05:09:24.443502 1840 webserver.cc:460] Webserver started at http://127.14.33.2:43359/ using document root <none> and password file <none>
I20260925 05:09:24.443758 1840 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:24.443831 1840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:24.443971 1840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:24.444996 1840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data/instance:
uuid: "443ce786dbd64ad88d49a1c316e36eb9"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "a77468373c3413951ab17ff61daaa119"
server_key_iv: "95abfe5265eefd2cb72e7e152d882ed8"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.445418 1840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance:
uuid: "443ce786dbd64ad88d49a1c316e36eb9"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "a77468373c3413951ab17ff61daaa119"
server_key_iv: "95abfe5265eefd2cb72e7e152d882ed8"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.446723 1840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:24.447460 1854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.447700 1840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:24.447799 1840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal
uuid: "443ce786dbd64ad88d49a1c316e36eb9"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "a77468373c3413951ab17ff61daaa119"
server_key_iv: "95abfe5265eefd2cb72e7e152d882ed8"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.447894 1840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:24.458577 1840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:24.458833 1840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:24.458979 1840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:24.459177 1840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:24.459491 1840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:24.459553 1840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.459591 1840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:24.459616 1840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.465466 1840 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.2:46563
I20260925 05:09:24.465528 1967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.2:46563 every 8 connection(s)
I20260925 05:09:24.465817 1840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/data/info.pb
I20260925 05:09:24.467907 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1840
I20260925 05:09:24.468039 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal/instance
I20260925 05:09:24.468250 14468 external_mini_cluster.cc:1428] Setting key 8d5e421d161e39bf309b55dc37808b33
I20260925 05:09:24.469255 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.33.3:0
--local_ip_for_outbound_sockets=127.14.33.3
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45461
--builtin_ntp_servers=127.14.33.20:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 05:09:24.471510 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.466502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.2:43337 (local address 127.14.33.62:45461)
0925 05:09:24.466616 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.466618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.466802 (+ 184us) server_negotiation.cc:407] Connection header received
0925 05:09:24.467277 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.467278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.467288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.467340 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:24.467991 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.468681 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.469930 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.470147 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.470246 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.471225 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:24.471227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:24.471227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:24.471233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:24.471249 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:24.471253 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:24.471313 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.471400 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.471427 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 05:09:24.471717 1970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.466465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45461 (local address 127.14.33.2:43337)
0925 05:09:24.466666 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.466705 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.467184 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.467401 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.467404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.467412 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:24.467899 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.467903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.468774 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.468775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.469810 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.469813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.470256 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.470258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.470403 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.471052 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:24.471053 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:24.471083 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:24.471104 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:24.471316 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:24.471321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:24.471324 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.471401 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.471500 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260925 05:09:24.472263 1968 heartbeater.cc:347] Connected to a master server at 127.14.33.62:45461
I20260925 05:09:24.472363 1968 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:24.472559 1968 heartbeater.cc:511] Master 127.14.33.62:45461 requested a full tablet report, sending...
I20260925 05:09:24.472960 1649 ts_manager.cc:194] Registered new tserver with Master: 443ce786dbd64ad88d49a1c316e36eb9 (127.14.33.2:46563)
I20260925 05:09:24.473536 1649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.2:43337
W20260925 05:09:24.550113 1971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:24.550379 1971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:24.550414 1971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:24.551550 1971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:24.551587 1971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:24.551612 1971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.33.3
I20260925 05:09:24.552840 1971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.33.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.33.3
--webserver_port=0
--tserver_master_addrs=127.14.33.62:45461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.33.3
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:24.552995 1971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:24.553222 1971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:24.557912 1977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:24.557934 1979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:24.558084 1971 server_base.cc:1034] running on GCE node
W20260925 05:09:24.557934 1976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:24.558401 1971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:24.558657 1971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:24.559821 1971 hybrid_clock.cc:648] HybridClock initialized: now 1790312964559802 us; error 31 us; skew 500 ppm
I20260925 05:09:24.560962 1971 webserver.cc:460] Webserver started at http://127.14.33.3:40989/ using document root <none> and password file <none>
I20260925 05:09:24.561247 1971 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:24.561328 1971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:24.561460 1971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:24.562436 1971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data/instance:
uuid: "487f42da02854b4b99082e8dcaff2520"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "613a87a8ca0a0d1b639a778c27047581"
server_key_iv: "f20d3c22b59d4aef089a5eb829c95bfc"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.562790 1971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance:
uuid: "487f42da02854b4b99082e8dcaff2520"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "613a87a8ca0a0d1b639a778c27047581"
server_key_iv: "f20d3c22b59d4aef089a5eb829c95bfc"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.564080 1971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:24.564883 1985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.565068 1971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:24.565188 1971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal
uuid: "487f42da02854b4b99082e8dcaff2520"
format_stamp: "Formatted at 2026-09-25 05:09:24 on dist-test-slave-4dmq"
server_key: "613a87a8ca0a0d1b639a778c27047581"
server_key_iv: "f20d3c22b59d4aef089a5eb829c95bfc"
server_key_version: "encryptionkey@0"
I20260925 05:09:24.565253 1971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:24.578720 1971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:24.578933 1971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:24.579025 1971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:24.579174 1971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 05:09:24.579470 1971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 05:09:24.579540 1971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.579609 1971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 05:09:24.579638 1971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:24.584947 1971 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.3:46589
I20260925 05:09:24.584980 2098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.3:46589 every 8 connection(s)
I20260925 05:09:24.585307 1971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/data/info.pb
I20260925 05:09:24.590229 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.586015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.3:37195 (local address 127.14.33.62:45461)
0925 05:09:24.586079 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.586081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.586270 (+ 189us) server_negotiation.cc:407] Connection header received
0925 05:09:24.586754 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.586756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.586768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.586824 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:24.587446 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.587941 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.588862 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.589054 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.589208 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.589944 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:24.589946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:24.589946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:24.589951 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:24.589965 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:24.589969 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:24.590016 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.590127 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.590154 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 05:09:24.590421 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.586004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:45461 (local address 127.14.33.3:37195)
0925 05:09:24.586161 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.586189 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.586647 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.586895 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.586899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.586906 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:24.587345 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.587349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.588041 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.588042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.588726 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.588727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.589186 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.589187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.589325 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.589803 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 05:09:24.589804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:24.589828 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 05:09:24.589848 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:24.590034 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:24.590039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:24.590042 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.590128 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.590218 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260925 05:09:24.590901 2099 heartbeater.cc:347] Connected to a master server at 127.14.33.62:45461
I20260925 05:09:24.591004 2099 heartbeater.cc:464] Registering TS with master...
I20260925 05:09:24.591178 2099 heartbeater.cc:511] Master 127.14.33.62:45461 requested a full tablet report, sending...
I20260925 05:09:24.591519 1649 ts_manager.cc:194] Registered new tserver with Master: 487f42da02854b4b99082e8dcaff2520 (127.14.33.3:46589)
I20260925 05:09:24.591841 1649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.33.3:37195
I20260925 05:09:24.595117 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 1971
I20260925 05:09:24.595206 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal/instance
I20260925 05:09:24.595360 14468 external_mini_cluster.cc:1428] Setting key 4b10ad82e020273149b05da60d2e5fab
I20260925 05:09:24.596299 14468 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 05:09:24.600020 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.597655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.14.33.62:45461)
0925 05:09:24.597687 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.597688 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.597836 (+ 148us) server_negotiation.cc:407] Connection header received
0925 05:09:24.597878 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.597878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.597886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.597922 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:24.598041 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.598432 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.599046 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.599198 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.599241 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.599761 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:24.599762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:24.599762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:24.599766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:24.599777 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:24.599780 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:24.599817 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.599868 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.599884 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 05:09:24.601341 1649 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 05:09:24.602725 1649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 05:09:24.608907 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.604696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:46589 (local address 127.0.0.1:42306)
0925 05:09:24.605428 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.605436 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.605522 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.605726 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.605726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.605729 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.605767 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.605768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.606870 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.606871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.607814 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.607816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.608663 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.608664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.608738 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.608748 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.608819 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.608856 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":4,"thread_start_us":102,"threads_started":1}
I20260925 05:09:24.608984 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.605027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.14.33.3:46589)
0925 05:09:24.605183 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.605187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.605519 (+ 332us) server_negotiation.cc:407] Connection header received
0925 05:09:24.605583 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.605585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.605599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.605671 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.605826 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.606806 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.607907 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.608558 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.608633 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.608709 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.608866 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.608910 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260925 05:09:24.609665 2033 tablet_service.cc:1467] Processing CreateTablet for tablet 4e99e42c79d94bcd9edbe5b54d11759e (DEFAULT_TABLE table=test-table [id=496817d009524eda85c0f44ef7f6607e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:24.610023 2033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e99e42c79d94bcd9edbe5b54d11759e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:24.610324 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.604913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:46563 (local address 127.0.0.1:34170)
0925 05:09:24.605605 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.605614 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.605655 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.605898 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.605899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.605902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.605931 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.605932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.606757 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.606758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.608029 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.608031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.610111 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.610112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.610188 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.610193 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.610250 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.610279 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":68,"thread_start_us":54,"threads_started":1}
I20260925 05:09:24.610401 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.605014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.14.33.2:46563)
0925 05:09:24.605216 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.605220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.605709 (+ 489us) server_negotiation.cc:407] Connection header received
0925 05:09:24.605749 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.605753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.605766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.605843 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.606014 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.606656 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.609205 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.609994 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.610092 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.610186 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.610263 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.610303 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260925 05:09:24.610980 1902 tablet_service.cc:1467] Processing CreateTablet for tablet 4e99e42c79d94bcd9edbe5b54d11759e (DEFAULT_TABLE table=test-table [id=496817d009524eda85c0f44ef7f6607e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:24.611289 1902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e99e42c79d94bcd9edbe5b54d11759e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:24.612838 2113 tablet_bootstrap.cc:492] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Bootstrap starting.
I20260925 05:09:24.613229 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.604696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:37733 (local address 127.0.0.1:58398)
0925 05:09:24.604850 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.604882 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.604966 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.605383 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.605385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.605386 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.605428 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.605429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.606805 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.606806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.608028 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.608031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.608980 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.608981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.609216 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.609238 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.613045 (+ 3807us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.613097 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":52,"thread_start_us":77,"threads_started":1}
I20260925 05:09:24.613238 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.604885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.14.33.1:37733)
0925 05:09:24.605224 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.605227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.605236 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:09:24.605270 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.605272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.605284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.605358 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.605613 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.606727 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.608116 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.608873 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.608942 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.609337 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.613091 (+ 3754us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.613157 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":73,"threads_started":1}
I20260925 05:09:24.613560 2113 tablet_bootstrap.cc:654] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:24.613893 2113 log.cc:824] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:24.614194 1771 tablet_service.cc:1467] Processing CreateTablet for tablet 4e99e42c79d94bcd9edbe5b54d11759e (DEFAULT_TABLE table=test-table [id=496817d009524eda85c0f44ef7f6607e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 05:09:24.614408 2115 tablet_bootstrap.cc:492] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Bootstrap starting.
I20260925 05:09:24.614533 1771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e99e42c79d94bcd9edbe5b54d11759e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:24.614969 2113 tablet_bootstrap.cc:492] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: No bootstrap required, opened a new log
I20260925 05:09:24.615039 2113 ts_tablet_manager.cc:1397] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 05:09:24.615060 2115 tablet_bootstrap.cc:654] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:24.615370 2115 log.cc:824] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:24.616150 2115 tablet_bootstrap.cc:492] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: No bootstrap required, opened a new log
I20260925 05:09:24.616205 2115 ts_tablet_manager.cc:1397] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:24.616976 2117 tablet_bootstrap.cc:492] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c: Bootstrap starting.
I20260925 05:09:24.617566 2117 tablet_bootstrap.cc:654] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:24.617769 2117 log.cc:824] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:24.617906 2113 raft_consensus.cc:348] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.618062 2113 raft_consensus.cc:374] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:24.618100 2113 raft_consensus.cc:729] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487f42da02854b4b99082e8dcaff2520, State: Initialized, Role: FOLLOWER
I20260925 05:09:24.618235 2113 consensus_queue.cc:260] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.618396 2117 tablet_bootstrap.cc:492] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c: No bootstrap required, opened a new log
I20260925 05:09:24.618484 2117 ts_tablet_manager.cc:1397] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 05:09:24.618490 2113 ts_tablet_manager.cc:1428] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 05:09:24.618563 2099 heartbeater.cc:503] Master 127.14.33.62:45461 was elected leader, sending a full tablet report...
I20260925 05:09:24.619098 2115 raft_consensus.cc:348] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.619256 2115 raft_consensus.cc:374] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:24.619299 2115 raft_consensus.cc:729] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443ce786dbd64ad88d49a1c316e36eb9, State: Initialized, Role: FOLLOWER
I20260925 05:09:24.619414 2115 consensus_queue.cc:260] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.619702 2115 ts_tablet_manager.cc:1428] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 05:09:24.619732 1968 heartbeater.cc:503] Master 127.14.33.62:45461 was elected leader, sending a full tablet report...
I20260925 05:09:24.621097 2117 raft_consensus.cc:348] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.621323 2117 raft_consensus.cc:374] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:24.621376 2117 raft_consensus.cc:729] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5ec4e485d34f8983000c3cc864142c, State: Initialized, Role: FOLLOWER
I20260925 05:09:24.621480 2117 consensus_queue.cc:260] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.621717 2117 ts_tablet_manager.cc:1428] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 05:09:24.621806 1837 heartbeater.cc:503] Master 127.14.33.62:45461 was elected leader, sending a full tablet report...
W20260925 05:09:24.716396 1969 tablet.cc:2369] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:24.727108 2121 raft_consensus.cc:482] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 05:09:24.727257 2121 raft_consensus.cc:504] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.727634 2121 leader_election.cc:288] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 487f42da02854b4b99082e8dcaff2520 (127.14.33.3:46589), 443ce786dbd64ad88d49a1c316e36eb9 (127.14.33.2:46563)
I20260925 05:09:24.731436 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.727828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.3:46589 (local address 127.14.33.1:57841)
0925 05:09:24.727887 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.727899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.727940 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.728153 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.728154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.728155 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.728226 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.728228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.729182 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.729182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.730428 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.730431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.731155 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.731156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.731231 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.731235 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.731310 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.731342 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":66}
I20260925 05:09:24.731451 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.727937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:57771 (local address 127.14.33.2:46563)
0925 05:09:24.727991 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.727993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.728108 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:09:24.728182 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.728183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.728191 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.728237 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.728366 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.729074 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.730527 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.731055 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.731128 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.731138 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.731335 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.731386 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 05:09:24.731451 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.727941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.33.1:57841 (local address 127.14.33.3:46589)
0925 05:09:24.727991 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.727993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.728002 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:09:24.728036 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.728036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.728044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.728137 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 05:09:24.728310 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.729081 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.730527 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.731058 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.731128 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.731138 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.731335 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.731381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:09:24.731441 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.727845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.2:46563 (local address 127.14.33.1:57771)
0925 05:09:24.728026 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.728041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:24.728120 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:24.728242 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:24.728242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:24.728244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 05:09:24.728301 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.728303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.729182 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.729182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.730428 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:24.730431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.731171 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.731171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:24.731236 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.731238 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 05:09:24.731297 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:09:24.731339 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":59,"thread_start_us":83,"threads_started":1}
I20260925 05:09:24.731967 1922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e99e42c79d94bcd9edbe5b54d11759e" candidate_uuid: "bf5ec4e485d34f8983000c3cc864142c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443ce786dbd64ad88d49a1c316e36eb9" is_pre_election: true
I20260925 05:09:24.731969 2053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e99e42c79d94bcd9edbe5b54d11759e" candidate_uuid: "bf5ec4e485d34f8983000c3cc864142c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487f42da02854b4b99082e8dcaff2520" is_pre_election: true
I20260925 05:09:24.732076 1922 raft_consensus.cc:2454] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5ec4e485d34f8983000c3cc864142c in term 0.
I20260925 05:09:24.732131 2053 raft_consensus.cc:2454] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5ec4e485d34f8983000c3cc864142c in term 0.
I20260925 05:09:24.732298 1726 leader_election.cc:302] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443ce786dbd64ad88d49a1c316e36eb9, bf5ec4e485d34f8983000c3cc864142c; no voters:
I20260925 05:09:24.732425 2121 raft_consensus.cc:2781] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 05:09:24.732476 2121 raft_consensus.cc:482] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 05:09:24.732499 2121 raft_consensus.cc:3037] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.733232 2121 raft_consensus.cc:504] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.733368 2121 leader_election.cc:288] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [CANDIDATE]: Term 1 election: Requested vote from peers 487f42da02854b4b99082e8dcaff2520 (127.14.33.3:46589), 443ce786dbd64ad88d49a1c316e36eb9 (127.14.33.2:46563)
I20260925 05:09:24.733502 2053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e99e42c79d94bcd9edbe5b54d11759e" candidate_uuid: "bf5ec4e485d34f8983000c3cc864142c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487f42da02854b4b99082e8dcaff2520"
I20260925 05:09:24.733510 1922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e99e42c79d94bcd9edbe5b54d11759e" candidate_uuid: "bf5ec4e485d34f8983000c3cc864142c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443ce786dbd64ad88d49a1c316e36eb9"
I20260925 05:09:24.733577 2053 raft_consensus.cc:3037] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.733603 1922 raft_consensus.cc:3037] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:24.734206 2053 raft_consensus.cc:2454] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5ec4e485d34f8983000c3cc864142c in term 1.
I20260925 05:09:24.734369 1922 raft_consensus.cc:2454] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5ec4e485d34f8983000c3cc864142c in term 1.
I20260925 05:09:24.734386 1725 leader_election.cc:302] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 487f42da02854b4b99082e8dcaff2520, bf5ec4e485d34f8983000c3cc864142c; no voters:
I20260925 05:09:24.734561 2121 raft_consensus.cc:2781] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:24.734709 2121 raft_consensus.cc:686] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [term 1 LEADER]: Becoming Leader. State: Replica: bf5ec4e485d34f8983000c3cc864142c, State: Running, Role: LEADER
I20260925 05:09:24.734807 2121 consensus_queue.cc:237] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } }
I20260925 05:09:24.735581 1649 catalog_manager.cc:5495] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5ec4e485d34f8983000c3cc864142c (127.14.33.1). New cstate: current_term: 1 leader_uuid: "bf5ec4e485d34f8983000c3cc864142c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf5ec4e485d34f8983000c3cc864142c" member_type: VOTER last_known_addr { host: "127.14.33.1" port: 37733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 } health_report { overall_health: UNKNOWN } } }
I20260925 05:09:24.752825 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.750575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.14.33.1:37733)
0925 05:09:24.750646 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:24.750647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:24.750655 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:24.750680 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:24.750681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:24.750695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:24.750790 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:24.750903 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.751301 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.752124 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:24.752285 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:24.752331 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:24.752506 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:24.752590 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:24.752646 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:24.752692 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 05:09:24.752717 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 05:09:24.754508 2053 raft_consensus.cc:1261] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 1 FOLLOWER]: Refusing update from remote peer bf5ec4e485d34f8983000c3cc864142c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:24.754508 1922 raft_consensus.cc:1261] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 1 FOLLOWER]: Refusing update from remote peer bf5ec4e485d34f8983000c3cc864142c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:24.754875 2123 consensus_queue.cc:1035] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [LEADER]: Connected to new peer: Peer: permanent_uuid: "487f42da02854b4b99082e8dcaff2520" member_type: VOTER last_known_addr { host: "127.14.33.3" port: 46589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:24.754983 2121 consensus_queue.cc:1035] T 4e99e42c79d94bcd9edbe5b54d11759e P bf5ec4e485d34f8983000c3cc864142c [LEADER]: Connected to new peer: Peer: permanent_uuid: "443ce786dbd64ad88d49a1c316e36eb9" member_type: VOTER last_known_addr { host: "127.14.33.2" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:24.757452 2128 mvcc.cc:205] Tried to move back new op lower bound from 7333121903631781888 to 7333121903554334720. Current Snapshot: MvccSnapshot[applied={T|T < 7333121903631781888}]
I20260925 05:09:24.757865 2127 mvcc.cc:205] Tried to move back new op lower bound from 7333121903631781888 to 7333121903554334720. Current Snapshot: MvccSnapshot[applied={T|T < 7333121903631781888 or (T in {7333121903631781888})}]
I20260925 05:09:24.759164 1648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:24.759253 1648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 05:09:24.760001 1648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba5b76dcaf3443ca955dbff3f5b25d4b: Sending DeleteTablet for 3 replicas of tablet 4e99e42c79d94bcd9edbe5b54d11759e
I20260925 05:09:24.760311 1902 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e99e42c79d94bcd9edbe5b54d11759e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:24 UTC) from {username='slave'} at 127.0.0.1:34170
I20260925 05:09:24.760529 2135 tablet_replica.cc:330] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: stopping tablet replica
I20260925 05:09:24.760526 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1709
I20260925 05:09:24.760532 1771 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e99e42c79d94bcd9edbe5b54d11759e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:24 UTC) from {username='slave'} at 127.0.0.1:58398
I20260925 05:09:24.760686 2135 raft_consensus.cc:2229] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:24.760807 2135 raft_consensus.cc:2258] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:24.761242 2135 ts_tablet_manager.cc:1905] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:24.761912 2033 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e99e42c79d94bcd9edbe5b54d11759e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 05:09:24 UTC) from {username='slave'} at 127.0.0.1:42306
I20260925 05:09:24.762116 2136 tablet_replica.cc:330] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: stopping tablet replica
I20260925 05:09:24.762200 2136 raft_consensus.cc:2229] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 05:09:24.762291 2136 raft_consensus.cc:2258] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 05:09:24.762676 2136 ts_tablet_manager.cc:1905] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 05:09:24.763039 2135 ts_tablet_manager.cc:1918] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:24.763123 2135 log.cc:1196] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-1/wal/wals/4e99e42c79d94bcd9edbe5b54d11759e
I20260925 05:09:24.763346 2135 ts_tablet_manager.cc:1939] T 4e99e42c79d94bcd9edbe5b54d11759e P 443ce786dbd64ad88d49a1c316e36eb9: Deleting consensus metadata
I20260925 05:09:24.763907 1635 catalog_manager.cc:4841] TS 443ce786dbd64ad88d49a1c316e36eb9 (127.14.33.2:46563): tablet 4e99e42c79d94bcd9edbe5b54d11759e (table test-table [id=496817d009524eda85c0f44ef7f6607e]) successfully deleted
I20260925 05:09:24.764832 2136 ts_tablet_manager.cc:1918] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 05:09:24.764904 2136 log.cc:1196] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Deleting WAL directory at /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312729583056-14468-0/minicluster-data/ts-2/wal/wals/4e99e42c79d94bcd9edbe5b54d11759e
I20260925 05:09:24.765154 2136 ts_tablet_manager.cc:1939] T 4e99e42c79d94bcd9edbe5b54d11759e P 487f42da02854b4b99082e8dcaff2520: Deleting consensus metadata
I20260925 05:09:24.765532 1634 catalog_manager.cc:4841] TS 487f42da02854b4b99082e8dcaff2520 (127.14.33.3:46589): tablet 4e99e42c79d94bcd9edbe5b54d11759e (table test-table [id=496817d009524eda85c0f44ef7f6607e]) successfully deleted
W20260925 05:09:24.766979 1635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.33.1:37733 (error 108)
I20260925 05:09:24.767292 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1840
I20260925 05:09:24.767390 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:24.767272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.1:37733 (local address 127.0.0.1:58422)
0925 05:09:24.767315 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:24.767355 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.1:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 05:09:24.769806 1635 catalog_manager.cc:4584] TS bf5ec4e485d34f8983000c3cc864142c (127.14.33.1:37733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e99e42c79d94bcd9edbe5b54d11759e: Network error: Client connection negotiation failed: client connection to 127.14.33.1:37733: connect: Connection refused (error 111)
I20260925 05:09:24.772787 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1971
I20260925 05:09:24.777338 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 1618
2026-09-25T05:09:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (729 ms)
[----------] 6 tests from EncryptionPolicyTest (8054 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:24 dist-test-slave-4dmq krb5kdc[2140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:24 dist-test-slave-4dmq krb5kdc[2140](info): set up 2 sockets
Sep 25 05:09:24 dist-test-slave-4dmq krb5kdc[2140](info): commencing operation
krb5kdc: starting...
W20260925 05:09:25.798229 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:25 dist-test-slave-4dmq krb5kdc[2140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:25.907459 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:25.983735 2153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:25.983959 2153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:25.983997 2153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:25.985080 2153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:25.985162 2153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:25.985186 2153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:25.985213 2153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:25.985239 2153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:25.986601 2153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:25.986866 2153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:25.987085 2153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:25.991734 2161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:25.991760 2159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:25.991827 2153 server_base.cc:1034] running on GCE node
W20260925 05:09:25.991760 2158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:25.992242 2153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:25.992551 2153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:25.993718 2153 hybrid_clock.cc:648] HybridClock initialized: now 1790312965993705 us; error 25 us; skew 500 ppm
Sep 25 05:09:25 dist-test-slave-4dmq krb5kdc[2140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:25.996034 2153 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:25.996454 2153 webserver.cc:460] Webserver started at http://127.0.0.1:42617/ using document root <none> and password file <none>
I20260925 05:09:25.996690 2153 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:25.996769 2153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:25.996946 2153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:25.997964 2153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "fedf1733d3624dcaa7b7f6af97a44f83"
format_stamp: "Formatted at 2026-09-25 05:09:25 on dist-test-slave-4dmq"
server_key: "e8603c785e6bac0a3a982b9cc38e1f80"
server_key_iv: "537970d122b319d7e44f94c27603c474"
server_key_version: "encryptionkey@0"
I20260925 05:09:25.998338 2153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "fedf1733d3624dcaa7b7f6af97a44f83"
format_stamp: "Formatted at 2026-09-25 05:09:25 on dist-test-slave-4dmq"
server_key: "e8603c785e6bac0a3a982b9cc38e1f80"
server_key_iv: "537970d122b319d7e44f94c27603c474"
server_key_version: "encryptionkey@0"
I20260925 05:09:25.999550 2153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:26.000329 2169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:26.000516 2153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:26.000612 2153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "fedf1733d3624dcaa7b7f6af97a44f83"
format_stamp: "Formatted at 2026-09-25 05:09:25 on dist-test-slave-4dmq"
server_key: "e8603c785e6bac0a3a982b9cc38e1f80"
server_key_iv: "537970d122b319d7e44f94c27603c474"
server_key_version: "encryptionkey@0"
I20260925 05:09:26.000689 2153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:26.012537 2153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:26.014286 2153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:26.014451 2153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:26.018356 2153 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43435
I20260925 05:09:26.018401 2221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43435 every 8 connection(s)
I20260925 05:09:26.018730 2153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:26.019532 2222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:26.021588 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2153
I20260925 05:09:26.021670 2222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83: Bootstrap starting.
I20260925 05:09:26.021745 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:26.021921 14468 external_mini_cluster.cc:1428] Setting key c24a16527441862010b201b6e9a435aa
I20260925 05:09:26.022330 2222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:26.022662 2222 log.cc:824] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:26.023329 2222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83: No bootstrap required, opened a new log
I20260925 05:09:26.025581 2222 raft_consensus.cc:348] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260925 05:09:26.025727 2222 raft_consensus.cc:374] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:26.025756 2222 raft_consensus.cc:729] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedf1733d3624dcaa7b7f6af97a44f83, State: Initialized, Role: FOLLOWER
I20260925 05:09:26.025866 2222 consensus_queue.cc:260] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260925 05:09:26.025951 2222 raft_consensus.cc:388] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:26.025980 2222 raft_consensus.cc:482] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:26.026033 2222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:26.026659 2222 raft_consensus.cc:504] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260925 05:09:26.026762 2222 leader_election.cc:302] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fedf1733d3624dcaa7b7f6af97a44f83; no voters:
I20260925 05:09:26.026918 2222 leader_election.cc:288] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:26.026969 2227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:26.027107 2227 raft_consensus.cc:686] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [term 1 LEADER]: Becoming Leader. State: Replica: fedf1733d3624dcaa7b7f6af97a44f83, State: Running, Role: LEADER
I20260925 05:09:26.027154 2222 sys_catalog.cc:564] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:26.027221 2227 consensus_queue.cc:237] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } }
I20260925 05:09:26.027557 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fedf1733d3624dcaa7b7f6af97a44f83. Latest consensus state: current_term: 1 leader_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } } }
I20260925 05:09:26.027726 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:26.027987 2236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:26.028026 2229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedf1733d3624dcaa7b7f6af97a44f83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43435 } } }
I20260925 05:09:26.028108 2229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:26.028847 2236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:09:26 dist-test-slave-4dmq krb5kdc[2140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:09:26.030604 2236 catalog_manager.cc:1347] Generated new cluster ID: ec7da684e8244d74b3219404d35dd2af
I20260925 05:09:26.030654 2236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:26.033095 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:26.022412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47729 (local address 127.0.0.1:43435)
0925 05:09:26.022608 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:26.022614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:26.022637 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:09:26.023220 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:26.023227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:26.023280 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:26.023420 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:26.024193 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:26.024995 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:26.026064 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:26.026680 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:26.026799 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:26.029711 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:26.029715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:26.029715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:26.029726 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:26.031410 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:26.031863 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:26.031866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:26.031866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:26.031973 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:26.032219 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:26.032220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:26.032220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:26.032509 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:26.032649 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:26.032810 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 05:09:26.032900 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260925 05:09:26.034080 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:26.042203 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:26.035383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56199 (local address 127.0.0.1:43435)
0925 05:09:26.035416 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:26.035418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:26.035501 (+ 83us) server_negotiation.cc:407] Connection header received
0925 05:09:26.035523 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:26.035524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:26.035532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:26.035591 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:26.036104 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:26.036838 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:26.037857 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:26.038164 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:26.038235 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:26.039502 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:26.039503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:26.039503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:26.039510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:26.040996 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:26.041389 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:26.041391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:26.041391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:26.041445 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:26.041658 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:26.041659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:26.041659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:26.041773 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:26.041883 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:26.041983 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:09:26.042039 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:09:26.045636 2236 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:26.046092 2236 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:26.053979 2236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fedf1733d3624dcaa7b7f6af97a44f83: Generated new TSK 0
I20260925 05:09:26.054147 2236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:26.058990 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2153
W20260925 05:09:26.062902 2138 connection.cc:535] client connection to 127.0.0.1:43435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43435): Connection reset by peer (error 104)
2026-09-25T05:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:26 dist-test-slave-4dmq krb5kdc[2254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:26 dist-test-slave-4dmq krb5kdc[2254](info): set up 2 sockets
Sep 25 05:09:26 dist-test-slave-4dmq krb5kdc[2254](info): commencing operation
krb5kdc: starting...
W20260925 05:09:27.049973 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:27 dist-test-slave-4dmq krb5kdc[2254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:27.160452 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:27.237735 2267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:27.237975 2267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:27.238014 2267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:27.239128 2267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:27.239207 2267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:27.239230 2267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:27.239259 2267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:27.239287 2267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:27.240726 2267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:27.241012 2267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:27.241269 2267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:27.246389 2274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:27.246405 2273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:27.246560 2267 server_base.cc:1034] running on GCE node
W20260925 05:09:27.246389 2276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:27.246850 2267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:27.247148 2267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:27.248318 2267 hybrid_clock.cc:648] HybridClock initialized: now 1790312967248305 us; error 46 us; skew 500 ppm
Sep 25 05:09:27 dist-test-slave-4dmq krb5kdc[2254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:27.254191 2267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:27.254671 2267 webserver.cc:460] Webserver started at http://127.0.0.1:35279/ using document root <none> and password file <none>
I20260925 05:09:27.254937 2267 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:27.255010 2267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:27.255162 2267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:27.256292 2267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "07a54fdf16634e3ab861cbbf1555dc52"
format_stamp: "Formatted at 2026-09-25 05:09:27 on dist-test-slave-4dmq"
server_key: "aaff16556d7f5d4da1d66c89eb9c81ce"
server_key_iv: "8f0f0d89bc9f9704af98bd9a8ec7bb4e"
server_key_version: "encryptionkey@0"
I20260925 05:09:27.256729 2267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "07a54fdf16634e3ab861cbbf1555dc52"
format_stamp: "Formatted at 2026-09-25 05:09:27 on dist-test-slave-4dmq"
server_key: "aaff16556d7f5d4da1d66c89eb9c81ce"
server_key_iv: "8f0f0d89bc9f9704af98bd9a8ec7bb4e"
server_key_version: "encryptionkey@0"
I20260925 05:09:27.258548 2267 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 05:09:27.259543 2283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:27.259747 2267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:27.259845 2267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "07a54fdf16634e3ab861cbbf1555dc52"
format_stamp: "Formatted at 2026-09-25 05:09:27 on dist-test-slave-4dmq"
server_key: "aaff16556d7f5d4da1d66c89eb9c81ce"
server_key_iv: "8f0f0d89bc9f9704af98bd9a8ec7bb4e"
server_key_version: "encryptionkey@0"
I20260925 05:09:27.259922 2267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:27.266933 2267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:27.269219 2267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:27.269326 2267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:27.273455 2267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36889
I20260925 05:09:27.273484 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36889 every 8 connection(s)
I20260925 05:09:27.273821 2267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:27.274611 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2267
I20260925 05:09:27.274734 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:27.274755 2336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:27.274952 14468 external_mini_cluster.cc:1428] Setting key 80d53c7f475577678bfc46a3c1b6abe4
I20260925 05:09:27.277855 2336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52: Bootstrap starting.
I20260925 05:09:27.278553 2336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:27.278883 2336 log.cc:824] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:27.279696 2336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52: No bootstrap required, opened a new log
I20260925 05:09:27.280681 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:27.275516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35523 (local address 127.0.0.1:36889)
0925 05:09:27.275681 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:27.275694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:27.275717 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:09:27.276189 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:27.276195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:27.276250 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:27.276367 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:27.276606 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:27.277526 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:27.278662 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:27.279126 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:27.279222 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:27.279950 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:27.279955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:27.279955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:27.280175 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:27.280258 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:27.280275 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:27.280372 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:27.280441 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 05:09:27.280469 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 05:09:27.281733 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:27.282403 2336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a54fdf16634e3ab861cbbf1555dc52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36889 } }
I20260925 05:09:27.282582 2336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:27.282614 2336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a54fdf16634e3ab861cbbf1555dc52, State: Initialized, Role: FOLLOWER
I20260925 05:09:27.282737 2336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a54fdf16634e3ab861cbbf1555dc52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36889 } }
I20260925 05:09:27.282831 2336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:27.282894 2336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:27.282948 2336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:27.283692 2336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a54fdf16634e3ab861cbbf1555dc52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36889 } }
I20260925 05:09:27.283802 2336 leader_election.cc:302] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a54fdf16634e3ab861cbbf1555dc52; no voters:
I20260925 05:09:27.283968 2336 leader_election.cc:288] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:27.284042 2346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:27.284180 2336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:27.284191 2346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [term 1 LEADER]: Becoming Leader. State: Replica: 07a54fdf16634e3ab861cbbf1555dc52, State: Running, Role: LEADER
I20260925 05:09:27.284301 2346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a54fdf16634e3ab861cbbf1555dc52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36889 } }
I20260925 05:09:27.284811 2346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a54fdf16634e3ab861cbbf1555dc52. Latest consensus state: current_term: 1 leader_uuid: "07a54fdf16634e3ab861cbbf1555dc52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a54fdf16634e3ab861cbbf1555dc52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36889 } } }
I20260925 05:09:27.284889 2346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:27.284818 2348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a54fdf16634e3ab861cbbf1555dc52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a54fdf16634e3ab861cbbf1555dc52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36889 } } }
I20260925 05:09:27.284929 2348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:27.285063 2352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:27.285569 2352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:27.286823 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:27.283181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36927 (local address 127.0.0.1:36889)
0925 05:09:27.283213 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:27.283215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:27.283353 (+ 138us) server_negotiation.cc:407] Connection header received
0925 05:09:27.283374 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:27.283374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:27.283383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:27.283448 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:27.283575 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:27.284301 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:27.285320 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:27.285657 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:27.285732 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:27.286525 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:27.286526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:27.286527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:27.286531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:27.286544 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:27.286548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:27.286593 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:27.286740 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 05:09:27.286749 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:27.287411 2352 catalog_manager.cc:1347] Generated new cluster ID: f57b8182ace245f8a49c2269f1b6e8e4
I20260925 05:09:27.287469 2352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:27.293320 2352 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:27.294005 2352 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:27.301033 2352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a54fdf16634e3ab861cbbf1555dc52: Generated new TSK 0
I20260925 05:09:27.301239 2352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:27.310961 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2267
W20260925 05:09:27.315207 2252 connection.cc:535] client connection to 127.0.0.1:36889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36889): Connection reset by peer (error 104)
2026-09-25T05:09:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:27 dist-test-slave-4dmq krb5kdc[2367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:27 dist-test-slave-4dmq krb5kdc[2367](info): set up 2 sockets
Sep 25 05:09:27 dist-test-slave-4dmq krb5kdc[2367](info): commencing operation
krb5kdc: starting...
W20260925 05:09:28.310081 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:28 dist-test-slave-4dmq krb5kdc[2367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:28.423838 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:28.501350 2380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:28.501555 2380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:28.501590 2380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:28.502645 2380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:28.502713 2380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:28.502738 2380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:28.502765 2380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:28.502784 2380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:28.504208 2380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:28.504474 2380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:28.504712 2380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:09:28.509307 2380 server_base.cc:1034] running on GCE node
W20260925 05:09:28.509292 2385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:28.509292 2386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:28.509284 2388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:28.509687 2380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:28.509920 2380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:28.511057 2380 hybrid_clock.cc:648] HybridClock initialized: now 1790312968511047 us; error 27 us; skew 500 ppm
Sep 25 05:09:28 dist-test-slave-4dmq krb5kdc[2367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:28.513371 2380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:28.513829 2380 webserver.cc:460] Webserver started at http://127.0.0.1:41525/ using document root <none> and password file <none>
I20260925 05:09:28.514065 2380 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:28.514137 2380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:28.514283 2380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:28.515244 2380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "45e451b31c014a50875e834589bc202a"
format_stamp: "Formatted at 2026-09-25 05:09:28 on dist-test-slave-4dmq"
server_key: "7811455e072c718e4d121810565e3aee"
server_key_iv: "0b6ea0a80243c5f0c7ee237affac037d"
server_key_version: "encryptionkey@0"
I20260925 05:09:28.515581 2380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "45e451b31c014a50875e834589bc202a"
format_stamp: "Formatted at 2026-09-25 05:09:28 on dist-test-slave-4dmq"
server_key: "7811455e072c718e4d121810565e3aee"
server_key_iv: "0b6ea0a80243c5f0c7ee237affac037d"
server_key_version: "encryptionkey@0"
I20260925 05:09:28.516745 2380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:09:28.517565 2395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:28.517741 2380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:28.517859 2380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "45e451b31c014a50875e834589bc202a"
format_stamp: "Formatted at 2026-09-25 05:09:28 on dist-test-slave-4dmq"
server_key: "7811455e072c718e4d121810565e3aee"
server_key_iv: "0b6ea0a80243c5f0c7ee237affac037d"
server_key_version: "encryptionkey@0"
I20260925 05:09:28.517935 2380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:28.520884 2380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:28.522574 2380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:28.522727 2380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:28.526657 2380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33277
I20260925 05:09:28.526685 2447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33277 every 8 connection(s)
I20260925 05:09:28.526978 2380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:28.527783 2448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:28.527786 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2380
I20260925 05:09:28.527892 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:28.528067 14468 external_mini_cluster.cc:1428] Setting key 523b6f742d065ba46738323a7c7410c4
I20260925 05:09:28.530375 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:28.528506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57711 (local address 127.0.0.1:33277)
0925 05:09:28.528691 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:28.528699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:28.528730 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:09:28.529192 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:28.529204 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:28.529227 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:28.529328 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:28.529940 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:28.529943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:28.529943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:28.529946 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:28.529977 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:28.529985 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:28.530036 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:28.530139 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 05:09:28.530166 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260925 05:09:28.530884 2448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a: Bootstrap starting.
I20260925 05:09:28.531301 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:28.531559 2448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:28.531850 2448 log.cc:824] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:28.532596 2448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a: No bootstrap required, opened a new log
I20260925 05:09:28.533610 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:28.532617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38535 (local address 127.0.0.1:33277)
0925 05:09:28.532652 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:28.532654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:28.532743 (+ 89us) server_negotiation.cc:407] Connection header received
0925 05:09:28.532790 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:28.532790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:28.532793 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:28.532837 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:28.533409 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:28.533409 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:28.533410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:28.533413 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:28.533423 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:28.533426 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:28.533469 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:28.533523 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 05:09:28.533531 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:28.534653 2448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e451b31c014a50875e834589bc202a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260925 05:09:28.534757 2448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:28.534789 2448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e451b31c014a50875e834589bc202a, State: Initialized, Role: FOLLOWER
I20260925 05:09:28.534904 2448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e451b31c014a50875e834589bc202a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260925 05:09:28.534988 2448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:28.535008 2448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:28.535032 2448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:28.535481 2448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e451b31c014a50875e834589bc202a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260925 05:09:28.535566 2448 leader_election.cc:302] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e451b31c014a50875e834589bc202a; no voters:
I20260925 05:09:28.535674 2448 leader_election.cc:288] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:28.535728 2458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:28.535826 2448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:28.535893 2458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [term 1 LEADER]: Becoming Leader. State: Replica: 45e451b31c014a50875e834589bc202a, State: Running, Role: LEADER
I20260925 05:09:28.535961 2458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e451b31c014a50875e834589bc202a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260925 05:09:28.536316 2459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e451b31c014a50875e834589bc202a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e451b31c014a50875e834589bc202a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } } }
I20260925 05:09:28.536412 2459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:28.536546 2460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e451b31c014a50875e834589bc202a. Latest consensus state: current_term: 1 leader_uuid: "45e451b31c014a50875e834589bc202a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e451b31c014a50875e834589bc202a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } } }
I20260925 05:09:28.536577 2467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:28.536603 2460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:28.537027 2467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:28.538467 2467 catalog_manager.cc:1347] Generated new cluster ID: 17556cfb26244983b172e0d0b9e619f8
I20260925 05:09:28.538516 2467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:28.561153 2467 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:28.561312 2467 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:28.565661 2467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e451b31c014a50875e834589bc202a: Generated new TSK 0
I20260925 05:09:28.565822 2467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:28.569363 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2380
2026-09-25T05:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:28 dist-test-slave-4dmq krb5kdc[2479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:28 dist-test-slave-4dmq krb5kdc[2479](info): set up 2 sockets
Sep 25 05:09:28 dist-test-slave-4dmq krb5kdc[2479](info): commencing operation
krb5kdc: starting...
W20260925 05:09:29.554109 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:29 dist-test-slave-4dmq krb5kdc[2479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:29.661063 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:29.736249 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:29.736426 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:29.736447 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:29.737491 2492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:29.737524 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:29.737537 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:29.737550 2492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:29.737560 2492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:29.738927 2492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:29.739110 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:29.739301 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:29.744210 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:29.744266 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:29.744196 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:29.744437 2492 server_base.cc:1034] running on GCE node
I20260925 05:09:29.744663 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:29.744921 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:29.746071 2492 hybrid_clock.cc:648] HybridClock initialized: now 1790312969746069 us; error 42 us; skew 500 ppm
Sep 25 05:09:29 dist-test-slave-4dmq krb5kdc[2479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:29.748451 2492 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:29.748885 2492 webserver.cc:460] Webserver started at http://127.0.0.1:33941/ using document root <none> and password file <none>
I20260925 05:09:29.749194 2492 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:29.749275 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:29.749410 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:29.750367 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "8aac39ef311d42caa26172ed4fd5c653"
format_stamp: "Formatted at 2026-09-25 05:09:29 on dist-test-slave-4dmq"
server_key: "040bb7150844a5e4a05dd7bb3eb02360"
server_key_iv: "36d93f7882706f8adfed9d4e9df486aa"
server_key_version: "encryptionkey@0"
I20260925 05:09:29.750720 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "8aac39ef311d42caa26172ed4fd5c653"
format_stamp: "Formatted at 2026-09-25 05:09:29 on dist-test-slave-4dmq"
server_key: "040bb7150844a5e4a05dd7bb3eb02360"
server_key_iv: "36d93f7882706f8adfed9d4e9df486aa"
server_key_version: "encryptionkey@0"
I20260925 05:09:29.751986 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:29.752696 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:29.752928 2492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:29.753036 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "8aac39ef311d42caa26172ed4fd5c653"
format_stamp: "Formatted at 2026-09-25 05:09:29 on dist-test-slave-4dmq"
server_key: "040bb7150844a5e4a05dd7bb3eb02360"
server_key_iv: "36d93f7882706f8adfed9d4e9df486aa"
server_key_version: "encryptionkey@0"
I20260925 05:09:29.753125 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:29.787554 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:29.789899 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:29.790069 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:29.794093 2492 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45155
I20260925 05:09:29.794128 2559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45155 every 8 connection(s)
I20260925 05:09:29.794446 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:29.795092 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2492
I20260925 05:09:29.795207 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:29.795307 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:29.795414 14468 external_mini_cluster.cc:1428] Setting key 2e219d3f226e8fce8a77fd91149a094a
I20260925 05:09:29.798561 2560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653: Bootstrap starting.
I20260925 05:09:29.799213 2560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:29.799523 2560 log.cc:824] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:29.800285 2560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653: No bootstrap required, opened a new log
Sep 25 05:09:29 dist-test-slave-4dmq krb5kdc[2479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:09:29.802824 2560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac39ef311d42caa26172ed4fd5c653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260925 05:09:29.802975 2560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:29.803018 2560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aac39ef311d42caa26172ed4fd5c653, State: Initialized, Role: FOLLOWER
I20260925 05:09:29.803159 2560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac39ef311d42caa26172ed4fd5c653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260925 05:09:29.803253 2560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:29.803287 2560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:29.803386 2560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:29.804083 2560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac39ef311d42caa26172ed4fd5c653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260925 05:09:29.804168 2560 leader_election.cc:302] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aac39ef311d42caa26172ed4fd5c653; no voters:
I20260925 05:09:29.804344 2560 leader_election.cc:288] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:29.804383 2565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:29.804513 2565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [term 1 LEADER]: Becoming Leader. State: Replica: 8aac39ef311d42caa26172ed4fd5c653, State: Running, Role: LEADER
I20260925 05:09:29.804628 2560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:29.804663 2565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac39ef311d42caa26172ed4fd5c653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } }
I20260925 05:09:29.805215 2566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aac39ef311d42caa26172ed4fd5c653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac39ef311d42caa26172ed4fd5c653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } } }
I20260925 05:09:29.805212 2565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aac39ef311d42caa26172ed4fd5c653. Latest consensus state: current_term: 1 leader_uuid: "8aac39ef311d42caa26172ed4fd5c653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aac39ef311d42caa26172ed4fd5c653" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45155 } } }
I20260925 05:09:29.805394 2566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:29.805418 2565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:29.805614 2574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:29.806166 2574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:29.806800 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:29.795907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42771 (local address 127.0.0.1:45155)
0925 05:09:29.796101 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:29.796112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:29.796144 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:09:29.796898 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:29.796907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:29.796971 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:29.797141 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:29.797740 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:29.799037 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:29.799970 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:29.800357 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:29.800483 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:29.802875 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:29.802879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:29.802879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:29.802889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:29.804846 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:29.805274 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:29.805277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:29.805277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:29.805383 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:29.805616 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:29.805617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:29.805617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:29.805923 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:29.806401 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:29.806469 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:09:29.806588 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260925 05:09:29.807760 2574 catalog_manager.cc:1347] Generated new cluster ID: 949ad5a4561c4ba5a436625139eafc9a
I20260925 05:09:29.807807 2574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:29.807972 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:29.815294 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:29.809205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58989 (local address 127.0.0.1:45155)
0925 05:09:29.809238 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:29.809240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:29.809376 (+ 136us) server_negotiation.cc:407] Connection header received
0925 05:09:29.809395 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:29.809396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:29.809405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:29.809448 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:29.809939 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:29.810393 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:29.811184 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:29.811656 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:29.811742 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:29.812852 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:29.812854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:29.812854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:29.812860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:29.814146 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:29.814522 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:29.814523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:29.814523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:29.814567 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:29.814807 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:29.814808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:29.814808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:29.814928 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:29.815042 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:29.815137 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 05:09:29.815203 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":16}
I20260925 05:09:29.832783 2574 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:29.833395 2574 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:29.839818 2574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aac39ef311d42caa26172ed4fd5c653: Generated new TSK 0
I20260925 05:09:29.840005 2574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:29.844013 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2492
2026-09-25T05:09:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:29 dist-test-slave-4dmq krb5kdc[2591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:29 dist-test-slave-4dmq krb5kdc[2591](info): set up 2 sockets
Sep 25 05:09:29 dist-test-slave-4dmq krb5kdc[2591](info): commencing operation
krb5kdc: starting...
W20260925 05:09:30.892149 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:30 dist-test-slave-4dmq krb5kdc[2591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:31.004243 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:31.080173 2604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:31.080396 2604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:31.080435 2604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:31.081610 2604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:31.081677 2604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:31.081702 2604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:31.081734 2604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:31.081756 2604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:31.083197 2604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:31.083509 2604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:31.083731 2604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:31.088624 2612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:31.088599 2609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:31.088733 2604 server_base.cc:1034] running on GCE node
W20260925 05:09:31.088660 2610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:31.089020 2604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:31.089334 2604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:31.090514 2604 hybrid_clock.cc:648] HybridClock initialized: now 1790312971090506 us; error 48 us; skew 500 ppm
Sep 25 05:09:31 dist-test-slave-4dmq krb5kdc[2591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:31.092851 2604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:31.093313 2604 webserver.cc:460] Webserver started at http://127.0.0.1:46279/ using document root <none> and password file <none>
I20260925 05:09:31.093531 2604 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:31.093603 2604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:31.093726 2604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:31.094753 2604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "9bd6f7822348479386f5761ae764c170"
format_stamp: "Formatted at 2026-09-25 05:09:31 on dist-test-slave-4dmq"
server_key: "7ef29e464d00e75e41889b1de3d2cf23"
server_key_iv: "9228509d6439d6b498e3fdb1ca8021da"
server_key_version: "encryptionkey@0"
I20260925 05:09:31.095101 2604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "9bd6f7822348479386f5761ae764c170"
format_stamp: "Formatted at 2026-09-25 05:09:31 on dist-test-slave-4dmq"
server_key: "7ef29e464d00e75e41889b1de3d2cf23"
server_key_iv: "9228509d6439d6b498e3fdb1ca8021da"
server_key_version: "encryptionkey@0"
I20260925 05:09:31.096349 2604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:31.097055 2619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:31.097245 2604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:31.097339 2604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "9bd6f7822348479386f5761ae764c170"
format_stamp: "Formatted at 2026-09-25 05:09:31 on dist-test-slave-4dmq"
server_key: "7ef29e464d00e75e41889b1de3d2cf23"
server_key_iv: "9228509d6439d6b498e3fdb1ca8021da"
server_key_version: "encryptionkey@0"
I20260925 05:09:31.097440 2604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:31.106076 2604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:31.107681 2604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:31.107831 2604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:31.111477 2604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35341
I20260925 05:09:31.111506 2671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35341 every 8 connection(s)
I20260925 05:09:31.111812 2604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:31.112594 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:31.114713 2672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170: Bootstrap starting.
I20260925 05:09:31.115245 2672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:31.115527 2672 log.cc:824] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:31.116145 2672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170: No bootstrap required, opened a new log
I20260925 05:09:31.117681 2672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6f7822348479386f5761ae764c170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260925 05:09:31.117800 2672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:31.117849 2672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd6f7822348479386f5761ae764c170, State: Initialized, Role: FOLLOWER
I20260925 05:09:31.118017 2672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6f7822348479386f5761ae764c170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260925 05:09:31.118078 2672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:31.118121 2672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:31.118170 2672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:31.118379 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2604
I20260925 05:09:31.118523 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:31.118697 14468 external_mini_cluster.cc:1428] Setting key 54d8b46c672acd746ba2b137c9f8e509
I20260925 05:09:31.118788 2672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6f7822348479386f5761ae764c170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260925 05:09:31.118923 2672 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd6f7822348479386f5761ae764c170; no voters:
I20260925 05:09:31.119072 2672 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:31.119158 2675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:31.119331 2675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd6f7822348479386f5761ae764c170, State: Running, Role: LEADER
I20260925 05:09:31.119398 2672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:31.119451 2675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6f7822348479386f5761ae764c170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } }
I20260925 05:09:31.119858 2677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd6f7822348479386f5761ae764c170. Latest consensus state: current_term: 1 leader_uuid: "9bd6f7822348479386f5761ae764c170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6f7822348479386f5761ae764c170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } } }
I20260925 05:09:31.119946 2677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:31.119978 2679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd6f7822348479386f5761ae764c170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6f7822348479386f5761ae764c170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35341 } } }
I20260925 05:09:31.120031 2679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:31.120461 2686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:31.121017 2686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:31.122642 2686 catalog_manager.cc:1347] Generated new cluster ID: c13d1c18ac73402d8922ef61568a40d8
I20260925 05:09:31.122691 2686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:31.125283 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:31.119155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58805 (local address 127.0.0.1:35341)
0925 05:09:31.119547 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:31.119628 (+ 81us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:31.119656 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:09:31.119740 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:31.119745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:31.119822 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:31.120268 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:31.120565 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:31.122058 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:31.123204 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:31.123852 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:31.123965 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:31.124605 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:31.124608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:31.124609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:31.124810 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:31.124846 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:31.124860 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:31.124930 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:31.125000 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 05:09:31.125059 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":92,"threads_started":1}
I20260925 05:09:31.126140 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:31.130803 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:31.127744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38331 (local address 127.0.0.1:35341)
0925 05:09:31.127798 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:31.127800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:31.127806 (+ 6us) server_negotiation.cc:407] Connection header received
0925 05:09:31.127822 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:31.127823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:31.127831 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:31.127878 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:31.128034 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:31.128734 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:31.129734 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:31.129990 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:31.130038 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:31.130610 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:31.130611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:31.130611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:31.130615 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:31.130626 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:31.130629 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:31.130672 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:31.130724 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 05:09:31.130743 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:09:31.143157 2686 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:31.143642 2686 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:31.146585 2686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd6f7822348479386f5761ae764c170: Generated new TSK 0
I20260925 05:09:31.146730 2686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:31.150827 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2604
2026-09-25T05:09:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:31 dist-test-slave-4dmq krb5kdc[2703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:31 dist-test-slave-4dmq krb5kdc[2703](info): set up 2 sockets
Sep 25 05:09:31 dist-test-slave-4dmq krb5kdc[2703](info): commencing operation
krb5kdc: starting...
W20260925 05:09:32.150138 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:32 dist-test-slave-4dmq krb5kdc[2703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:32.264031 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:32.347168 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:32.347375 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:32.347399 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:32.348611 2716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:32.348660 2716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:32.348676 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:32.348699 2716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:32.348762 2716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:32.350492 2716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:32.350773 2716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:32.351006 2716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:32.355942 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:32.355952 2716 server_base.cc:1034] running on GCE node
W20260925 05:09:32.355966 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:32.355952 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:32.356485 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:32.356807 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:32.357969 2716 hybrid_clock.cc:648] HybridClock initialized: now 1790312972357955 us; error 44 us; skew 500 ppm
Sep 25 05:09:32 dist-test-slave-4dmq krb5kdc[2703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:32.366370 2716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:32.366729 2716 webserver.cc:460] Webserver started at http://127.0.0.1:43835/ using document root <none> and password file <none>
I20260925 05:09:32.366905 2716 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:32.366945 2716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:32.367035 2716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:32.367985 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "e7c4bd717d35485ca314a02b73e874ff"
format_stamp: "Formatted at 2026-09-25 05:09:32 on dist-test-slave-4dmq"
server_key: "4648c901f6eb49af89b18db7c5fdac79"
server_key_iv: "8d0f1e3db145704dbc8a02fe23198cbf"
server_key_version: "encryptionkey@0"
I20260925 05:09:32.368279 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "e7c4bd717d35485ca314a02b73e874ff"
format_stamp: "Formatted at 2026-09-25 05:09:32 on dist-test-slave-4dmq"
server_key: "4648c901f6eb49af89b18db7c5fdac79"
server_key_iv: "8d0f1e3db145704dbc8a02fe23198cbf"
server_key_version: "encryptionkey@0"
I20260925 05:09:32.369494 2716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:32.370172 2731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:32.370370 2716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:32.370456 2716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "e7c4bd717d35485ca314a02b73e874ff"
format_stamp: "Formatted at 2026-09-25 05:09:32 on dist-test-slave-4dmq"
server_key: "4648c901f6eb49af89b18db7c5fdac79"
server_key_iv: "8d0f1e3db145704dbc8a02fe23198cbf"
server_key_version: "encryptionkey@0"
I20260925 05:09:32.370551 2716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:32.373872 2716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:32.376240 2716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:32.376385 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:32.380074 2716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43093
I20260925 05:09:32.380136 2783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43093 every 8 connection(s)
I20260925 05:09:32.380409 2716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:32.381217 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:32.383123 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff: Bootstrap starting.
I20260925 05:09:32.383601 2784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:32.383809 2784 log.cc:824] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:32.384382 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff: No bootstrap required, opened a new log
I20260925 05:09:32.385811 2784 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c4bd717d35485ca314a02b73e874ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260925 05:09:32.385928 2784 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:32.385948 2784 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c4bd717d35485ca314a02b73e874ff, State: Initialized, Role: FOLLOWER
I20260925 05:09:32.386065 2784 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c4bd717d35485ca314a02b73e874ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260925 05:09:32.386132 2784 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:32.386152 2784 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:32.386178 2784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:32.386593 2784 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c4bd717d35485ca314a02b73e874ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260925 05:09:32.386684 2784 leader_election.cc:302] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c4bd717d35485ca314a02b73e874ff; no voters:
I20260925 05:09:32.386821 2784 leader_election.cc:288] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:32.386888 2787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:32.387053 2787 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [term 1 LEADER]: Becoming Leader. State: Replica: e7c4bd717d35485ca314a02b73e874ff, State: Running, Role: LEADER
I20260925 05:09:32.387074 2784 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:32.387169 2787 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c4bd717d35485ca314a02b73e874ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260925 05:09:32.387535 2789 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c4bd717d35485ca314a02b73e874ff. Latest consensus state: current_term: 1 leader_uuid: "e7c4bd717d35485ca314a02b73e874ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c4bd717d35485ca314a02b73e874ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } } }
I20260925 05:09:32.387530 2788 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c4bd717d35485ca314a02b73e874ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c4bd717d35485ca314a02b73e874ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } } }
I20260925 05:09:32.387624 2789 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:32.387629 2788 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:32.387851 2791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:32.388171 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2716
I20260925 05:09:32.388321 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:32.388509 14468 external_mini_cluster.cc:1428] Setting key 6c62e32bdcc16385a39ba79defd78653
I20260925 05:09:32.388811 2791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:32.390188 2791 catalog_manager.cc:1347] Generated new cluster ID: fcac5feea2bd4f29bd6af6ea465bfd20
I20260925 05:09:32.390239 2791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:32.398808 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:32.389105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36145 (local address 127.0.0.1:43093)
0925 05:09:32.397256 (+ 8151us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:32.397266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:32.397300 (+ 34us) server_negotiation.cc:407] Connection header received
0925 05:09:32.397423 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:32.397430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:32.397473 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:32.397871 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:32.398345 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:32.398348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:32.398348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:32.398352 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:32.398385 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:32.398397 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:32.398460 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:32.398577 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 05:09:32.398618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8079,"thread_start_us":69,"threads_started":1}
I20260925 05:09:32.399655 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:32.402190 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:32.401177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50979 (local address 127.0.0.1:43093)
0925 05:09:32.401213 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:32.401215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:32.401337 (+ 122us) server_negotiation.cc:407] Connection header received
0925 05:09:32.401390 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:32.401391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:32.401396 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:32.401454 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:32.401994 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:32.401995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:32.401995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:32.401999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:32.402013 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:32.402017 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:32.402075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:32.402122 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 05:09:32.402133 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 05:09:32.419759 2791 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:32.419926 2791 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:32.423697 2791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c4bd717d35485ca314a02b73e874ff: Generated new TSK 0
I20260925 05:09:32.423848 2791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:32.431463 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2716
2026-09-25T05:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:32 dist-test-slave-4dmq krb5kdc[2815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:32 dist-test-slave-4dmq krb5kdc[2815](info): set up 2 sockets
Sep 25 05:09:32 dist-test-slave-4dmq krb5kdc[2815](info): commencing operation
krb5kdc: starting...
W20260925 05:09:33.422151 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:33 dist-test-slave-4dmq krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:33.531505 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:33.606860 2828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:33.607091 2828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:33.607124 2828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:33.608168 2828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:33.608232 2828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:33.608256 2828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:33.608285 2828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:33.608305 2828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:33.609725 2828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:33.609977 2828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:33.610183 2828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:33.614672 2833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:33.614683 2836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:33.614682 2834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:33.614941 2828 server_base.cc:1034] running on GCE node
I20260925 05:09:33.615180 2828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:33.615483 2828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:33.616616 2828 hybrid_clock.cc:648] HybridClock initialized: now 1790312973616602 us; error 28 us; skew 500 ppm
Sep 25 05:09:33 dist-test-slave-4dmq krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:33.619012 2828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:33.619419 2828 webserver.cc:460] Webserver started at http://127.0.0.1:40519/ using document root <none> and password file <none>
I20260925 05:09:33.619590 2828 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:33.619625 2828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:33.619704 2828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:33.620553 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "3f94b1c4710142c99cd7e749a7761c3e"
format_stamp: "Formatted at 2026-09-25 05:09:33 on dist-test-slave-4dmq"
server_key: "d001713e8b74dfb6b7afde1d668c66ac"
server_key_iv: "910712b9d95655d784fa4c416dc94721"
server_key_version: "encryptionkey@0"
I20260925 05:09:33.620829 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "3f94b1c4710142c99cd7e749a7761c3e"
format_stamp: "Formatted at 2026-09-25 05:09:33 on dist-test-slave-4dmq"
server_key: "d001713e8b74dfb6b7afde1d668c66ac"
server_key_iv: "910712b9d95655d784fa4c416dc94721"
server_key_version: "encryptionkey@0"
I20260925 05:09:33.622078 2828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:09:33.622728 2844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:33.622890 2828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:33.622953 2828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "3f94b1c4710142c99cd7e749a7761c3e"
format_stamp: "Formatted at 2026-09-25 05:09:33 on dist-test-slave-4dmq"
server_key: "d001713e8b74dfb6b7afde1d668c66ac"
server_key_iv: "910712b9d95655d784fa4c416dc94721"
server_key_version: "encryptionkey@0"
I20260925 05:09:33.623059 2828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:33.652876 2828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:33.655161 2828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:33.655288 2828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:33.659276 2828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42053
I20260925 05:09:33.659340 2896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42053 every 8 connection(s)
I20260925 05:09:33.659654 2828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:33.660418 2897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:33.662516 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e: Bootstrap starting.
I20260925 05:09:33.663048 2897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:33.663311 2897 log.cc:824] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:33.664006 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e: No bootstrap required, opened a new log
I20260925 05:09:33.665491 2897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f94b1c4710142c99cd7e749a7761c3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42053 } }
I20260925 05:09:33.665633 2897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:33.665671 2897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f94b1c4710142c99cd7e749a7761c3e, State: Initialized, Role: FOLLOWER
I20260925 05:09:33.665776 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2828
I20260925 05:09:33.665807 2897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f94b1c4710142c99cd7e749a7761c3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42053 } }
I20260925 05:09:33.665880 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:33.665898 2897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:33.665930 2897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:33.665968 2897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:33.666055 14468 external_mini_cluster.cc:1428] Setting key fa2b5b14a15ef59c9d85f4374ca64c86
I20260925 05:09:33.666596 2897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f94b1c4710142c99cd7e749a7761c3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42053 } }
I20260925 05:09:33.666707 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f94b1c4710142c99cd7e749a7761c3e; no voters:
I20260925 05:09:33.666875 2897 leader_election.cc:288] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:33.666923 2902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:33.667109 2902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [term 1 LEADER]: Becoming Leader. State: Replica: 3f94b1c4710142c99cd7e749a7761c3e, State: Running, Role: LEADER
I20260925 05:09:33.667124 2897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:33.667295 2902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f94b1c4710142c99cd7e749a7761c3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42053 } }
I20260925 05:09:33.667863 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f94b1c4710142c99cd7e749a7761c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f94b1c4710142c99cd7e749a7761c3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42053 } } }
I20260925 05:09:33.667951 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:33.668108 2903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f94b1c4710142c99cd7e749a7761c3e. Latest consensus state: current_term: 1 leader_uuid: "3f94b1c4710142c99cd7e749a7761c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f94b1c4710142c99cd7e749a7761c3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42053 } } }
I20260925 05:09:33.668164 2903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:33.668183 2911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:33.668723 2911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:33.670365 2911 catalog_manager.cc:1347] Generated new cluster ID: 353b8cf8652244b1ad0c3e1d5396a740
I20260925 05:09:33.670414 2911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 05:09:33 dist-test-slave-4dmq krb5kdc[2815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:09:33.676805 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:33.666504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:59829 (local address 127.0.0.1:42053)
0925 05:09:33.666660 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:33.666669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:33.666700 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:09:33.666814 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:33.666825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:33.666898 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:33.667035 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:33.667745 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:33.669169 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:33.670186 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:33.670720 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:33.670814 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:33.673327 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:33.673331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:33.673331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:33.673340 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:33.675108 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:33.675560 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:33.675562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:33.675562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:33.675661 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:33.676027 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:33.676027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:33.676028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:33.676274 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:33.676412 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:33.676562 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 05:09:33.676643 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260925 05:09:33.677726 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:33.685740 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:33.679088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:34469 (local address 127.0.0.1:42053)
0925 05:09:33.679116 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:33.679117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:33.679204 (+ 87us) server_negotiation.cc:407] Connection header received
0925 05:09:33.679253 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:33.679254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:33.679261 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:33.679302 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:33.679874 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:33.680548 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:33.681545 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:33.681824 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:33.681875 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:33.683078 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:33.683079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:33.683079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:33.683085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:33.684511 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:33.684939 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:33.684940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:33.684940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:33.684989 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:33.685241 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:33.685241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:33.685242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:33.685355 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:33.685440 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:33.685594 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 05:09:33.685649 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 05:09:33.687177 2911 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:33.687583 2911 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:33.689648 2911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f94b1c4710142c99cd7e749a7761c3e: Generated new TSK 0
I20260925 05:09:33.689793 2911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:33.696743 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2828
W20260925 05:09:33.700971 2813 connection.cc:535] client connection to 127.0.0.1:42053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42053): Connection reset by peer (error 104)
2026-09-25T05:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:33 dist-test-slave-4dmq krb5kdc[2929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:33 dist-test-slave-4dmq krb5kdc[2929](info): set up 2 sockets
Sep 25 05:09:33 dist-test-slave-4dmq krb5kdc[2929](info): commencing operation
krb5kdc: starting...
W20260925 05:09:34.726459 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:34 dist-test-slave-4dmq krb5kdc[2929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:34.847918 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:34.927194 2942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:34.927393 2942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:34.927415 2942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:34.928535 2942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:34.928576 2942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:34.928591 2942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:34.928603 2942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:34.928615 2942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:34.930006 2942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:34.930191 2942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:34.930347 2942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:34.935339 2948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:34.935343 2951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:34.935338 2949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:34.935518 2942 server_base.cc:1034] running on GCE node
I20260925 05:09:34.935801 2942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:34.936040 2942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:34.937187 2942 hybrid_clock.cc:648] HybridClock initialized: now 1790312974937179 us; error 40 us; skew 500 ppm
Sep 25 05:09:34 dist-test-slave-4dmq krb5kdc[2929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:34.939524 2942 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:34.939965 2942 webserver.cc:460] Webserver started at http://127.0.0.1:34569/ using document root <none> and password file <none>
I20260925 05:09:34.940146 2942 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:34.940183 2942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:34.940266 2942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:34.941291 2942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "13011d5e2b674a41b3558fc44450d13f"
format_stamp: "Formatted at 2026-09-25 05:09:34 on dist-test-slave-4dmq"
server_key: "fe85331f1fa18e57c022dd25fc885dda"
server_key_iv: "beaccc8ab88c4a79ec3a7b6659a1b620"
server_key_version: "encryptionkey@0"
I20260925 05:09:34.941589 2942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "13011d5e2b674a41b3558fc44450d13f"
format_stamp: "Formatted at 2026-09-25 05:09:34 on dist-test-slave-4dmq"
server_key: "fe85331f1fa18e57c022dd25fc885dda"
server_key_iv: "beaccc8ab88c4a79ec3a7b6659a1b620"
server_key_version: "encryptionkey@0"
I20260925 05:09:34.942785 2942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:34.943488 2958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:34.943691 2942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:34.943774 2942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "13011d5e2b674a41b3558fc44450d13f"
format_stamp: "Formatted at 2026-09-25 05:09:34 on dist-test-slave-4dmq"
server_key: "fe85331f1fa18e57c022dd25fc885dda"
server_key_iv: "beaccc8ab88c4a79ec3a7b6659a1b620"
server_key_version: "encryptionkey@0"
I20260925 05:09:34.943835 2942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:34.957854 2942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:34.960048 2942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:34.960151 2942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:34.963975 2942 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44267
I20260925 05:09:34.964049 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44267 every 8 connection(s)
I20260925 05:09:34.964395 2942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:34.965284 3011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:34.967527 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f: Bootstrap starting.
I20260925 05:09:34.968096 3011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:34.968387 3011 log.cc:824] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:34.969067 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f: No bootstrap required, opened a new log
I20260925 05:09:34.970641 3011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13011d5e2b674a41b3558fc44450d13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44267 } }
I20260925 05:09:34.970794 3011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:34.970842 3011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13011d5e2b674a41b3558fc44450d13f, State: Initialized, Role: FOLLOWER
I20260925 05:09:34.971012 3011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13011d5e2b674a41b3558fc44450d13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44267 } }
I20260925 05:09:34.971113 3011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:34.971156 3011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:34.971206 3011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:34.971719 3011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13011d5e2b674a41b3558fc44450d13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44267 } }
I20260925 05:09:34.971832 3011 leader_election.cc:302] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13011d5e2b674a41b3558fc44450d13f; no voters:
I20260925 05:09:34.972003 3011 leader_election.cc:288] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:34.972043 3014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:34.972245 3011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:34.972434 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 2942
I20260925 05:09:34.972584 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:34.972769 14468 external_mini_cluster.cc:1428] Setting key d4af1935358ba47dea08f70fd6a277f0
I20260925 05:09:34.972230 3014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [term 1 LEADER]: Becoming Leader. State: Replica: 13011d5e2b674a41b3558fc44450d13f, State: Running, Role: LEADER
I20260925 05:09:34.973026 3014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13011d5e2b674a41b3558fc44450d13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44267 } }
I20260925 05:09:34.973883 3016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13011d5e2b674a41b3558fc44450d13f. Latest consensus state: current_term: 1 leader_uuid: "13011d5e2b674a41b3558fc44450d13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13011d5e2b674a41b3558fc44450d13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44267 } } }
I20260925 05:09:34.973730 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13011d5e2b674a41b3558fc44450d13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13011d5e2b674a41b3558fc44450d13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44267 } } }
I20260925 05:09:34.974018 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:34.974198 3016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:34.974291 3030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:34.974377 3030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:34.974475 3033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:34.974670 3033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:34.976068 3033 catalog_manager.cc:1347] Generated new cluster ID: c4ab71f967724427a93f38d856962bc9
I20260925 05:09:34.976118 3033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:34.979221 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:34.973316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:45117 (local address 127.0.0.1:44267)
0925 05:09:34.973964 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:34.973974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:34.974010 (+ 36us) server_negotiation.cc:407] Connection header received
0925 05:09:34.974132 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:34.974140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:34.974209 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:34.974377 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:34.974690 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:34.976026 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:34.977159 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:34.977731 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:34.977846 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:34.978562 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:34.978566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:34.978566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:34.978764 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:34.978798 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:34.978821 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:34.978896 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:34.979010 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 05:09:34.979034 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":102,"threads_started":1}
I20260925 05:09:34.980152 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:34.988890 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:34.985423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:52231 (local address 127.0.0.1:44267)
0925 05:09:34.985465 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:34.985466 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:34.985610 (+ 144us) server_negotiation.cc:407] Connection header received
0925 05:09:34.985666 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:34.985666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:34.985677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:34.985727 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:34.985852 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:34.986559 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:34.987568 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:34.987876 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:34.987952 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:34.988627 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:34.988629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:34.988629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:34.988634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:34.988648 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:34.988652 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:34.988709 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:34.988800 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 05:09:34.988817 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:09:35.001261 3033 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:35.001845 3033 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:35.004856 3033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13011d5e2b674a41b3558fc44450d13f: Generated new TSK 0
I20260925 05:09:35.005053 3033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:35.008955 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 2942
W20260925 05:09:35.013082 2927 connection.cc:535] client connection to 127.0.0.1:44267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44267): Connection reset by peer (error 104)
2026-09-25T05:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:35 dist-test-slave-4dmq krb5kdc[3042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:35 dist-test-slave-4dmq krb5kdc[3042](info): set up 2 sockets
Sep 25 05:09:35 dist-test-slave-4dmq krb5kdc[3042](info): commencing operation
krb5kdc: starting...
W20260925 05:09:35.994087 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:36 dist-test-slave-4dmq krb5kdc[3042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:36.105813 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:36.182468 3055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:36.182698 3055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:36.182734 3055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:36.183866 3055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:36.183938 3055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:36.183960 3055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:36.183990 3055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:36.184018 3055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:36.185648 3055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:36.185909 3055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:36.186123 3055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:36.190800 3063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:36.190869 3060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:36.190886 3055 server_base.cc:1034] running on GCE node
W20260925 05:09:36.190824 3061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:36.191326 3055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:36.191700 3055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:36.192878 3055 hybrid_clock.cc:648] HybridClock initialized: now 1790312976192856 us; error 39 us; skew 500 ppm
Sep 25 05:09:36 dist-test-slave-4dmq krb5kdc[3042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:36.195541 3055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:36.195951 3055 webserver.cc:460] Webserver started at http://127.0.0.1:40421/ using document root <none> and password file <none>
I20260925 05:09:36.196169 3055 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:36.196245 3055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:36.196412 3055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:36.197476 3055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "97802c435af04fd8a43b367446c6210a"
format_stamp: "Formatted at 2026-09-25 05:09:36 on dist-test-slave-4dmq"
server_key: "3d55a73eec42cdf22fa76c6b1531fcc9"
server_key_iv: "6b68670486c1ec18d8318a53248cf2f4"
server_key_version: "encryptionkey@0"
I20260925 05:09:36.197813 3055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "97802c435af04fd8a43b367446c6210a"
format_stamp: "Formatted at 2026-09-25 05:09:36 on dist-test-slave-4dmq"
server_key: "3d55a73eec42cdf22fa76c6b1531fcc9"
server_key_iv: "6b68670486c1ec18d8318a53248cf2f4"
server_key_version: "encryptionkey@0"
I20260925 05:09:36.199079 3055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 05:09:36.199749 3070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:36.199918 3055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:36.200013 3055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "97802c435af04fd8a43b367446c6210a"
format_stamp: "Formatted at 2026-09-25 05:09:36 on dist-test-slave-4dmq"
server_key: "3d55a73eec42cdf22fa76c6b1531fcc9"
server_key_iv: "6b68670486c1ec18d8318a53248cf2f4"
server_key_version: "encryptionkey@0"
I20260925 05:09:36.200098 3055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:36.203245 3055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:36.205045 3055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:36.205238 3055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:36.208997 3055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41739
I20260925 05:09:36.209065 3122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41739 every 8 connection(s)
I20260925 05:09:36.209436 3055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:36.209661 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3055
I20260925 05:09:36.209803 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:36.209968 14468 external_mini_cluster.cc:1428] Setting key 177f8d14c668e7d8058d46413f1bd6e3
I20260925 05:09:36.210448 3123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:36.212397 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:36.210419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:33253 (local address 127.0.0.1:41739)
0925 05:09:36.210584 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:36.210591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:36.210733 (+ 142us) server_negotiation.cc:407] Connection header received
0925 05:09:36.211219 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:36.211228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:36.211258 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:36.211408 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:36.211967 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:36.211971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:36.211971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:36.211975 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:36.212019 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:36.212031 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:36.212104 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:36.212166 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 05:09:36.212206 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260925 05:09:36.213378 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:36.213742 3123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a: Bootstrap starting.
I20260925 05:09:36.214352 3123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:36.214702 3123 log.cc:824] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:36.215384 3123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a: No bootstrap required, opened a new log
I20260925 05:09:36.215708 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:36.214731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:50235 (local address 127.0.0.1:41739)
0925 05:09:36.214764 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:36.214765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:36.214849 (+ 84us) server_negotiation.cc:407] Connection header received
0925 05:09:36.214917 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:36.214917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:36.214920 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:36.214990 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:36.215525 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:36.215527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:36.215527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:36.215531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:36.215543 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:36.215546 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:36.215604 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:36.215642 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 05:09:36.215653 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:09:36.217314 3123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97802c435af04fd8a43b367446c6210a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260925 05:09:36.217453 3123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:36.217494 3123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97802c435af04fd8a43b367446c6210a, State: Initialized, Role: FOLLOWER
I20260925 05:09:36.217602 3123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97802c435af04fd8a43b367446c6210a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260925 05:09:36.217686 3123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:36.217726 3123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:36.217768 3123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:36.218250 3123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97802c435af04fd8a43b367446c6210a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260925 05:09:36.218367 3123 leader_election.cc:302] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97802c435af04fd8a43b367446c6210a; no voters:
I20260925 05:09:36.218544 3123 leader_election.cc:288] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:36.218575 3133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:36.218685 3133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [term 1 LEADER]: Becoming Leader. State: Replica: 97802c435af04fd8a43b367446c6210a, State: Running, Role: LEADER
I20260925 05:09:36.218777 3123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:36.218812 3133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97802c435af04fd8a43b367446c6210a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } }
I20260925 05:09:36.219084 3133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97802c435af04fd8a43b367446c6210a. Latest consensus state: current_term: 1 leader_uuid: "97802c435af04fd8a43b367446c6210a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97802c435af04fd8a43b367446c6210a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } } }
I20260925 05:09:36.219168 3133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:36.219094 3135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97802c435af04fd8a43b367446c6210a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97802c435af04fd8a43b367446c6210a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41739 } } }
I20260925 05:09:36.219236 3135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:36.219383 3139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:36.219874 3139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:36.221055 3150 mvcc.cc:205] Tried to move back new op lower bound from 7333121950600011776 to 7333121950592561152. Current Snapshot: MvccSnapshot[applied={T|T < 7333121950600011776}]
I20260925 05:09:36.221484 3139 catalog_manager.cc:1347] Generated new cluster ID: d5bec8982e1048a5a0238b87e57d4782
I20260925 05:09:36.221530 3139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:36.226765 3139 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:36.226924 3139 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:36.235673 3139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97802c435af04fd8a43b367446c6210a: Generated new TSK 0
I20260925 05:09:36.235870 3139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:36.240929 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3055
2026-09-25T05:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:36 dist-test-slave-4dmq krb5kdc[3154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:36 dist-test-slave-4dmq krb5kdc[3154](info): set up 2 sockets
Sep 25 05:09:36 dist-test-slave-4dmq krb5kdc[3154](info): commencing operation
krb5kdc: starting...
W20260925 05:09:37.229955 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:37 dist-test-slave-4dmq krb5kdc[3154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:37.341791 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:37.418812 3167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:37.418980 3167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:37.419001 3167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:37.419934 3167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:37.419966 3167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:37.419978 3167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:37.419991 3167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:37.420001 3167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:37.421366 3167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:37.421538 3167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:37.421715 3167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:37.426260 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:37.426297 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:37.426281 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:37.426406 3167 server_base.cc:1034] running on GCE node
I20260925 05:09:37.426688 3167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:37.426987 3167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:37.428143 3167 hybrid_clock.cc:648] HybridClock initialized: now 1790312977428113 us; error 47 us; skew 500 ppm
Sep 25 05:09:37 dist-test-slave-4dmq krb5kdc[3154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:37.433965 3167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:37.434314 3167 webserver.cc:460] Webserver started at http://127.0.0.1:41499/ using document root <none> and password file <none>
I20260925 05:09:37.434492 3167 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:37.434526 3167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:37.434602 3167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:37.435511 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "68750ce931c440ec8d82752d092ec441"
format_stamp: "Formatted at 2026-09-25 05:09:37 on dist-test-slave-4dmq"
server_key: "d18117cc88cd5e6fd4300f85e7d92dd1"
server_key_iv: "7659d110678360cf527af749e0daf573"
server_key_version: "encryptionkey@0"
I20260925 05:09:37.435801 3167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "68750ce931c440ec8d82752d092ec441"
format_stamp: "Formatted at 2026-09-25 05:09:37 on dist-test-slave-4dmq"
server_key: "d18117cc88cd5e6fd4300f85e7d92dd1"
server_key_iv: "7659d110678360cf527af749e0daf573"
server_key_version: "encryptionkey@0"
I20260925 05:09:37.436945 3167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:37.437655 3182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:37.437911 3167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 05:09:37.437971 3167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "68750ce931c440ec8d82752d092ec441"
format_stamp: "Formatted at 2026-09-25 05:09:37 on dist-test-slave-4dmq"
server_key: "d18117cc88cd5e6fd4300f85e7d92dd1"
server_key_iv: "7659d110678360cf527af749e0daf573"
server_key_version: "encryptionkey@0"
I20260925 05:09:37.438057 3167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:37.454411 3167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:37.457376 3167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:37.457530 3167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:37.462522 3167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37777
I20260925 05:09:37.462574 3234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37777 every 8 connection(s)
I20260925 05:09:37.463001 3167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:37.464082 3235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:37.465799 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3167
I20260925 05:09:37.466135 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:37.466348 14468 external_mini_cluster.cc:1428] Setting key fbab3de6a2e77445fe1a25afcdf307fb
I20260925 05:09:37.467872 3235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441: Bootstrap starting.
I20260925 05:09:37.468586 3235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:37.468942 3235 log.cc:824] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:37.469748 3235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441: No bootstrap required, opened a new log
I20260925 05:09:37.472406 3235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68750ce931c440ec8d82752d092ec441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37777 } }
I20260925 05:09:37.472574 3235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:37.472661 3235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68750ce931c440ec8d82752d092ec441, State: Initialized, Role: FOLLOWER
I20260925 05:09:37.472846 3235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68750ce931c440ec8d82752d092ec441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37777 } }
I20260925 05:09:37.472944 3235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:37.472977 3235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:37.473083 3235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:37.473866 3235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68750ce931c440ec8d82752d092ec441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37777 } }
I20260925 05:09:37.474002 3235 leader_election.cc:302] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68750ce931c440ec8d82752d092ec441; no voters:
I20260925 05:09:37.474151 3235 leader_election.cc:288] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:37.474174 3240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:37.474377 3240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [term 1 LEADER]: Becoming Leader. State: Replica: 68750ce931c440ec8d82752d092ec441, State: Running, Role: LEADER
I20260925 05:09:37.474475 3235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:37.474496 3240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68750ce931c440ec8d82752d092ec441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37777 } }
Sep 25 05:09:37 dist-test-slave-4dmq krb5kdc[3154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:09:37.474864 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68750ce931c440ec8d82752d092ec441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68750ce931c440ec8d82752d092ec441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37777 } } }
I20260925 05:09:37.474937 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:37.475119 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68750ce931c440ec8d82752d092ec441. Latest consensus state: current_term: 1 leader_uuid: "68750ce931c440ec8d82752d092ec441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68750ce931c440ec8d82752d092ec441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37777 } } }
I20260925 05:09:37.475240 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:37.475389 3247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:37.475947 3247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:37.479329 3247 catalog_manager.cc:1347] Generated new cluster ID: a43be727520d46578b64f5fad0e6ec02
I20260925 05:09:37.479386 3247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:37.480877 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:37.466866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:46525 (local address 127.0.0.1:37777)
0925 05:09:37.467022 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:37.467031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:37.467059 (+ 28us) server_negotiation.cc:407] Connection header received
0925 05:09:37.467540 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:37.467547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:37.467593 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:37.467766 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:37.468589 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:37.470129 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:37.471250 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:37.472033 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:37.472196 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:37.475566 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:37.475572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:37.475572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:37.475585 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:37.478634 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:37.479099 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:37.479102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:37.479103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:37.479224 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:37.479493 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:37.479493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:37.479494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:37.479770 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:37.479891 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:37.480510 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 05:09:37.480640 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260925 05:09:37.482019 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:37.498732 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:37.485385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:41597 (local address 127.0.0.1:37777)
0925 05:09:37.485431 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:37.485433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:37.491785 (+ 6352us) server_negotiation.cc:407] Connection header received
0925 05:09:37.491814 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:37.491815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:37.491826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:37.491897 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:37.492563 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:37.493317 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:37.494418 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:37.494721 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:37.494756 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:37.496117 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:37.496118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:37.496119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:37.496125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:37.497604 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:37.497963 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:37.497964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:37.497964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:37.498012 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:37.498215 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:37.498216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:37.498217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:37.498318 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:37.498382 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:37.498591 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 05:09:37.498647 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 05:09:37.500309 3247 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:37.501183 3247 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:37.505779 3247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68750ce931c440ec8d82752d092ec441: Generated new TSK 0
I20260925 05:09:37.505964 3247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:37.509573 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3167
W20260925 05:09:37.514969 3152 connection.cc:535] client connection to 127.0.0.1:37777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37777): Connection reset by peer (error 104)
2026-09-25T05:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:37 dist-test-slave-4dmq krb5kdc[3266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:37 dist-test-slave-4dmq krb5kdc[3266](info): set up 2 sockets
Sep 25 05:09:37 dist-test-slave-4dmq krb5kdc[3266](info): commencing operation
krb5kdc: starting...
W20260925 05:09:38.482328 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:38 dist-test-slave-4dmq krb5kdc[3266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:38.590090 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:38.666958 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:38.667174 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:38.667209 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:38.668310 3279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:38.668372 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:38.668397 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:38.668426 3279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:38.668448 3279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:38.669996 3279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:38.670252 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:38.670461 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:38.674909 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:38.674914 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:38.675135 3279 server_base.cc:1034] running on GCE node
W20260925 05:09:38.674954 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:38.675437 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:38.675698 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:38.676859 3279 hybrid_clock.cc:648] HybridClock initialized: now 1790312978676846 us; error 25 us; skew 500 ppm
Sep 25 05:09:38 dist-test-slave-4dmq krb5kdc[3266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:38.679147 3279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:38.679615 3279 webserver.cc:460] Webserver started at http://127.0.0.1:40719/ using document root <none> and password file <none>
I20260925 05:09:38.679831 3279 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:38.679893 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:38.680016 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:38.680970 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "1cf10e4562724035b9eb3b632a65cdb5"
format_stamp: "Formatted at 2026-09-25 05:09:38 on dist-test-slave-4dmq"
server_key: "4fef42b9d422e776fa0c2eb115866ab4"
server_key_iv: "970a21e829d82f9a358b4ad24d2653b4"
server_key_version: "encryptionkey@0"
I20260925 05:09:38.681331 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "1cf10e4562724035b9eb3b632a65cdb5"
format_stamp: "Formatted at 2026-09-25 05:09:38 on dist-test-slave-4dmq"
server_key: "4fef42b9d422e776fa0c2eb115866ab4"
server_key_iv: "970a21e829d82f9a358b4ad24d2653b4"
server_key_version: "encryptionkey@0"
I20260925 05:09:38.682660 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:38.683374 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:38.683568 3279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:38.683658 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "1cf10e4562724035b9eb3b632a65cdb5"
format_stamp: "Formatted at 2026-09-25 05:09:38 on dist-test-slave-4dmq"
server_key: "4fef42b9d422e776fa0c2eb115866ab4"
server_key_iv: "970a21e829d82f9a358b4ad24d2653b4"
server_key_version: "encryptionkey@0"
I20260925 05:09:38.683732 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:38.700704 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:38.702886 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:38.703043 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:38.707026 3279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45365
I20260925 05:09:38.707068 3346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45365 every 8 connection(s)
I20260925 05:09:38.707404 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:38.708235 3347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:38.710464 3347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5: Bootstrap starting.
I20260925 05:09:38.711000 3347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:38.711303 3347 log.cc:824] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:38.711947 3347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5: No bootstrap required, opened a new log
I20260925 05:09:38.713601 3347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf10e4562724035b9eb3b632a65cdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45365 } }
I20260925 05:09:38.713745 3347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:38.713768 3347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf10e4562724035b9eb3b632a65cdb5, State: Initialized, Role: FOLLOWER
I20260925 05:09:38.713934 3347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf10e4562724035b9eb3b632a65cdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45365 } }
I20260925 05:09:38.713994 3347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:38.714047 3347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:38.714097 3347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:38.714208 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3279
I20260925 05:09:38.714341 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:38.714525 14468 external_mini_cluster.cc:1428] Setting key 65c56893fe08cd5cd026049b3fac409e
I20260925 05:09:38.714740 3347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf10e4562724035b9eb3b632a65cdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45365 } }
I20260925 05:09:38.714855 3347 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf10e4562724035b9eb3b632a65cdb5; no voters:
I20260925 05:09:38.715076 3347 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:38.715098 3351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:38.715207 3351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf10e4562724035b9eb3b632a65cdb5, State: Running, Role: LEADER
I20260925 05:09:38.715286 3351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf10e4562724035b9eb3b632a65cdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45365 } }
I20260925 05:09:38.715343 3347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:38.715641 3353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf10e4562724035b9eb3b632a65cdb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf10e4562724035b9eb3b632a65cdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45365 } } }
I20260925 05:09:38.715760 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf10e4562724035b9eb3b632a65cdb5. Latest consensus state: current_term: 1 leader_uuid: "1cf10e4562724035b9eb3b632a65cdb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf10e4562724035b9eb3b632a65cdb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45365 } } }
I20260925 05:09:38.715795 3353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:38.715873 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:38.716368 3360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:38.716878 3360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:38.718668 3360 catalog_manager.cc:1347] Generated new cluster ID: ec4db64f058c47fcb36092369ed103ef
I20260925 05:09:38.718729 3360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:38.720835 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:38.714971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:40001 (local address 127.0.0.1:45365)
0925 05:09:38.716091 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:38.716101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:38.716133 (+ 32us) server_negotiation.cc:407] Connection header received
0925 05:09:38.716244 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:38.716250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:38.716313 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:38.716754 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:38.716847 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:38.718120 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:38.719031 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:38.719443 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:38.719550 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:38.720222 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:38.720226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:38.720226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:38.720400 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:38.720428 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:38.720439 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:38.720529 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:38.720607 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 05:09:38.720634 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":85,"threads_started":1}
I20260925 05:09:38.721886 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:38.724408 3360 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:38.725039 3360 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:38.727758 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:38.724312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:40279 (local address 127.0.0.1:45365)
0925 05:09:38.724341 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:38.724343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:38.725276 (+ 933us) server_negotiation.cc:407] Connection header received
0925 05:09:38.725346 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:38.725347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:38.725366 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:38.725416 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:38.725526 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:38.725987 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:38.726673 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:38.726865 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:38.726940 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:38.727527 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:38.727528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:38.727528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:38.727532 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:38.727544 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:38.727564 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:38.727626 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:38.727682 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 05:09:38.727693 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:09:38.730518 3360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf10e4562724035b9eb3b632a65cdb5: Generated new TSK 0
I20260925 05:09:38.730666 3360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:38.734299 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3279
W20260925 05:09:38.738089 3264 connection.cc:535] client connection to 127.0.0.1:45365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45365): Connection reset by peer (error 104)
2026-09-25T05:09:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:38 dist-test-slave-4dmq krb5kdc[3378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:38 dist-test-slave-4dmq krb5kdc[3378](info): set up 2 sockets
Sep 25 05:09:38 dist-test-slave-4dmq krb5kdc[3378](info): commencing operation
krb5kdc: starting...
W20260925 05:09:39.718382 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:39 dist-test-slave-4dmq krb5kdc[3378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:09:39.825625 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:39.899205 3391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:39.899430 3391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:39.899471 3391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:39.900599 3391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:39.900667 3391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:39.900691 3391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:39.900712 3391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:39.900743 3391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:39.902160 3391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:39.902418 3391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:39.902628 3391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:09:39.907280 3391 server_base.cc:1034] running on GCE node
W20260925 05:09:39.907246 3396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:39.907262 3399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:39.907258 3397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:39.907716 3391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:39.908006 3391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:39.909183 3391 hybrid_clock.cc:648] HybridClock initialized: now 1790312979909166 us; error 25 us; skew 500 ppm
Sep 25 05:09:39 dist-test-slave-4dmq krb5kdc[3378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:39.911413 3391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:09:39.911839 3391 webserver.cc:460] Webserver started at http://127.0.0.1:36657/ using document root <none> and password file <none>
I20260925 05:09:39.912057 3391 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:39.912120 3391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:39.912254 3391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:39.913268 3391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "78d0aafc7e34405c914c80c2ec439147"
format_stamp: "Formatted at 2026-09-25 05:09:39 on dist-test-slave-4dmq"
server_key: "bb0f975da67958d4eeddd1faaf22789a"
server_key_iv: "c41018d64d6df421393e4c50cd80925f"
server_key_version: "encryptionkey@0"
I20260925 05:09:39.913597 3391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "78d0aafc7e34405c914c80c2ec439147"
format_stamp: "Formatted at 2026-09-25 05:09:39 on dist-test-slave-4dmq"
server_key: "bb0f975da67958d4eeddd1faaf22789a"
server_key_iv: "c41018d64d6df421393e4c50cd80925f"
server_key_version: "encryptionkey@0"
I20260925 05:09:39.914876 3391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:39.915635 3406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:39.915807 3391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:39.915898 3391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "78d0aafc7e34405c914c80c2ec439147"
format_stamp: "Formatted at 2026-09-25 05:09:39 on dist-test-slave-4dmq"
server_key: "bb0f975da67958d4eeddd1faaf22789a"
server_key_iv: "c41018d64d6df421393e4c50cd80925f"
server_key_version: "encryptionkey@0"
I20260925 05:09:39.915972 3391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:39.919183 3391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:39.920998 3391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:39.921187 3391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:39.924935 3391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32875
I20260925 05:09:39.924963 3458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32875 every 8 connection(s)
I20260925 05:09:39.925324 3391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:39.926093 3459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:39.928072 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147: Bootstrap starting.
I20260925 05:09:39.928609 3459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:39.928871 3459 log.cc:824] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:39.929540 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3391
I20260925 05:09:39.929574 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147: No bootstrap required, opened a new log
I20260925 05:09:39.929662 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:39.929845 14468 external_mini_cluster.cc:1428] Setting key 9125bd778c5372fec4f7fbd0850852b0
I20260925 05:09:39.931697 3459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0aafc7e34405c914c80c2ec439147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260925 05:09:39.931840 3459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:39.931911 3459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d0aafc7e34405c914c80c2ec439147, State: Initialized, Role: FOLLOWER
I20260925 05:09:39.932031 3459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0aafc7e34405c914c80c2ec439147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260925 05:09:39.932124 3459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:39.932181 3459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:39.932228 3459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:39.932924 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:39.930345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:56103 (local address 127.0.0.1:32875)
0925 05:09:39.930517 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:39.930524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:39.930547 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:09:39.931693 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:39.931701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:39.931770 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:39.931914 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:39.932527 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:39.932530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:39.932530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:39.932534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:39.932579 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:39.932589 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:39.932651 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:39.932715 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 05:09:39.932756 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260925 05:09:39.933229 3459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0aafc7e34405c914c80c2ec439147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260925 05:09:39.933332 3459 leader_election.cc:302] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d0aafc7e34405c914c80c2ec439147; no voters:
I20260925 05:09:39.933488 3459 leader_election.cc:288] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:39.933521 3464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:39.933691 3464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [term 1 LEADER]: Becoming Leader. State: Replica: 78d0aafc7e34405c914c80c2ec439147, State: Running, Role: LEADER
I20260925 05:09:39.933700 3459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:39.933843 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:39.933799 3464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0aafc7e34405c914c80c2ec439147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } }
I20260925 05:09:39.934309 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d0aafc7e34405c914c80c2ec439147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0aafc7e34405c914c80c2ec439147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } } }
I20260925 05:09:39.934614 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:39.934538 3466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d0aafc7e34405c914c80c2ec439147. Latest consensus state: current_term: 1 leader_uuid: "78d0aafc7e34405c914c80c2ec439147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d0aafc7e34405c914c80c2ec439147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32875 } } }
I20260925 05:09:39.935053 3466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147 [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:39.935633 3484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:39.935685 3484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:39.935734 3479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:39.936143 3479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:39.936379 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:39.935422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.128:59183 (local address 127.0.0.1:32875)
0925 05:09:39.935451 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:39.935453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:39.935502 (+ 49us) server_negotiation.cc:407] Connection header received
0925 05:09:39.935574 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:39.935575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:39.935580 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:39.935633 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:39.936144 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:39.936146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:39.936146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:39.936150 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:39.936162 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:39.936166 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:39.936229 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:39.936291 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 05:09:39.936304 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:09:39.937633 3479 catalog_manager.cc:1347] Generated new cluster ID: 964b330bd6b54471ab45ca8e9c970007
I20260925 05:09:39.937683 3479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:39.954154 3479 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:39.954319 3479 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:39.956802 3479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d0aafc7e34405c914c80c2ec439147: Generated new TSK 0
I20260925 05:09:39.956974 3479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:39.958530 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3391
2026-09-25T05:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:40 dist-test-slave-4dmq krb5kdc[3490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:40 dist-test-slave-4dmq krb5kdc[3490](info): set up 2 sockets
Sep 25 05:09:40 dist-test-slave-4dmq krb5kdc[3490](info): commencing operation
krb5kdc: starting...
W20260925 05:09:40.978307 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:41 dist-test-slave-4dmq krb5kdc[3490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:41.087936 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42617
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:42617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:41.165729 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:41.165920 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:41.165942 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:41.167063 3503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:41.167104 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:41.167119 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:41.167155 3503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:41.167182 3503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:41.168574 3503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:42617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:42617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:41.168817 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:41.169032 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:41.173597 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:41.173744 3503 server_base.cc:1034] running on GCE node
W20260925 05:09:41.173784 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:41.173611 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:41.174152 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:41.174490 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:41.175647 3503 hybrid_clock.cc:648] HybridClock initialized: now 1790312981175649 us; error 45 us; skew 500 ppm
Sep 25 05:09:41 dist-test-slave-4dmq krb5kdc[3490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312981, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:41.178016 3503 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:41.178469 3503 webserver.cc:460] Webserver started at http://127.14.33.62:44055/ using document root <none> and password file <none>
I20260925 05:09:41.178702 3503 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:41.178771 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:41.178929 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:41.179956 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "29a226e12cea4a7dbdc2686b1148587f"
format_stamp: "Formatted at 2026-09-25 05:09:41 on dist-test-slave-4dmq"
server_key: "49669040b5f16e11d3764d775a629008"
server_key_iv: "95e31eda095e87912d1eb4218cf1e5ea"
server_key_version: "encryptionkey@0"
I20260925 05:09:41.180321 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "29a226e12cea4a7dbdc2686b1148587f"
format_stamp: "Formatted at 2026-09-25 05:09:41 on dist-test-slave-4dmq"
server_key: "49669040b5f16e11d3764d775a629008"
server_key_iv: "95e31eda095e87912d1eb4218cf1e5ea"
server_key_version: "encryptionkey@0"
I20260925 05:09:41.181730 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:41.182449 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:41.182615 3503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:41.182703 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "29a226e12cea4a7dbdc2686b1148587f"
format_stamp: "Formatted at 2026-09-25 05:09:41 on dist-test-slave-4dmq"
server_key: "49669040b5f16e11d3764d775a629008"
server_key_iv: "95e31eda095e87912d1eb4218cf1e5ea"
server_key_version: "encryptionkey@0"
I20260925 05:09:41.182791 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:41.192015 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:41.193903 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:41.194057 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:41.198040 3503 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:42617
I20260925 05:09:41.198087 3571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:42617 every 8 connection(s)
I20260925 05:09:41.198534 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:41.199340 3572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:41.201706 3572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f: Bootstrap starting.
I20260925 05:09:41.201967 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3503
I20260925 05:09:41.202106 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:41.202299 14468 external_mini_cluster.cc:1428] Setting key 634cba6a9fdb443bf95c675d7048ba22
I20260925 05:09:41.202457 3572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:41.202790 3572 log.cc:824] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:41.203476 3572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f: No bootstrap required, opened a new log
I20260925 05:09:41.205835 3572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a226e12cea4a7dbdc2686b1148587f" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42617 } }
I20260925 05:09:41.206000 3572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:41.206034 3572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a226e12cea4a7dbdc2686b1148587f, State: Initialized, Role: FOLLOWER
I20260925 05:09:41.206144 3572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a226e12cea4a7dbdc2686b1148587f" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42617 } }
I20260925 05:09:41.206208 3572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:41.206252 3572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:41.206295 3572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:41.206877 3572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a226e12cea4a7dbdc2686b1148587f" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42617 } }
I20260925 05:09:41.207000 3572 leader_election.cc:302] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29a226e12cea4a7dbdc2686b1148587f; no voters:
I20260925 05:09:41.207167 3572 leader_election.cc:288] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:41.207188 3577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:41.207293 3577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [term 1 LEADER]: Becoming Leader. State: Replica: 29a226e12cea4a7dbdc2686b1148587f, State: Running, Role: LEADER
I20260925 05:09:41.207371 3577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a226e12cea4a7dbdc2686b1148587f" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42617 } }
I20260925 05:09:41.207461 3572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:41.207659 3577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29a226e12cea4a7dbdc2686b1148587f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a226e12cea4a7dbdc2686b1148587f" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42617 } } }
I20260925 05:09:41.207768 3577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:41.207690 3578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29a226e12cea4a7dbdc2686b1148587f. Latest consensus state: current_term: 1 leader_uuid: "29a226e12cea4a7dbdc2686b1148587f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a226e12cea4a7dbdc2686b1148587f" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 42617 } } }
I20260925 05:09:41.208006 3578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:41.209234 3593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:41.209291 3593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:41.209345 3582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 05:09:41 dist-test-slave-4dmq krb5kdc[3490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:41.209506 3582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:41.211066 3582 catalog_manager.cc:1347] Generated new cluster ID: a4558e84373b4ecdacc7b19f5593bf4b
I20260925 05:09:41.211119 3582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:41.213243 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:41.202839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41475 (local address 127.14.33.62:42617)
0925 05:09:41.203017 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:41.203027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:41.203060 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:09:41.203562 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:41.203592 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:41.203651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:41.203792 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:41.204584 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:41.205489 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:41.206564 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:41.206955 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:41.207073 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:41.209906 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:41.209910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:41.209911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:41.209922 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:41.211706 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:41.212083 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:41.212086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:41.212087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:41.212178 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:41.212417 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:41.212417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:41.212418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:41.212634 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:41.212821 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:41.212910 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 05:09:41.213007 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260925 05:09:41.214075 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:41.220799 3582 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:41.221266 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:41.215351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39249 (local address 127.14.33.62:42617)
0925 05:09:41.215384 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:41.215385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:41.215393 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:41.215441 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:41.215442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:41.215450 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:41.215498 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:41.215946 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:41.216413 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:41.217082 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:41.217398 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:41.217465 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:41.218645 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:41.218646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:41.218646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:41.218653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:41.219940 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:41.220288 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:41.220289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:41.220289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:41.220343 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:41.220707 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:41.220708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:41.220709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:41.220817 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:41.220905 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:41.221088 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 05:09:41.221165 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:09:41.221419 3582 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:41.229185 3582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29a226e12cea4a7dbdc2686b1148587f: Generated new TSK 0
I20260925 05:09:41.229337 3582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:41.231863 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3503
W20260925 05:09:41.235976 3488 connection.cc:535] client connection to 127.14.33.62:42617 recv error: Network error: failed to read from TLS socket (remote: 127.14.33.62:42617): Connection reset by peer (error 104)
2026-09-25T05:09:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:41 dist-test-slave-4dmq krb5kdc[3604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:41 dist-test-slave-4dmq krb5kdc[3604](info): set up 2 sockets
Sep 25 05:09:41 dist-test-slave-4dmq krb5kdc[3604](info): commencing operation
krb5kdc: starting...
W20260925 05:09:42.242074 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:42 dist-test-slave-4dmq krb5kdc[3604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:42.353704 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38629
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:42.432492 3617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:42.432736 3617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:42.432778 3617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:42.433948 3617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:42.434015 3617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:42.434039 3617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:42.434084 3617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:42.434116 3617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:42.435542 3617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:38629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:42.435807 3617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:42.436030 3617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:42.440598 3624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:42.440598 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:42.440579 3626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:42.440781 3617 server_base.cc:1034] running on GCE node
I20260925 05:09:42.440984 3617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:42.441269 3617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:42.442411 3617 hybrid_clock.cc:648] HybridClock initialized: now 1790312982442401 us; error 27 us; skew 500 ppm
Sep 25 05:09:42 dist-test-slave-4dmq krb5kdc[3604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312982, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:42.449246 3617 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:42.449720 3617 webserver.cc:460] Webserver started at http://127.14.33.62:46793/ using document root <none> and password file <none>
I20260925 05:09:42.449947 3617 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:42.449990 3617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:42.450133 3617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:42.451143 3617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "d8090ca574774263a0c1ba2f8cb1c9a1"
format_stamp: "Formatted at 2026-09-25 05:09:42 on dist-test-slave-4dmq"
server_key: "355e320fb8effc9f1120e3921692c315"
server_key_iv: "a2c74b1fd153879a7de4a2417f67b5ba"
server_key_version: "encryptionkey@0"
I20260925 05:09:42.451515 3617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "d8090ca574774263a0c1ba2f8cb1c9a1"
format_stamp: "Formatted at 2026-09-25 05:09:42 on dist-test-slave-4dmq"
server_key: "355e320fb8effc9f1120e3921692c315"
server_key_iv: "a2c74b1fd153879a7de4a2417f67b5ba"
server_key_version: "encryptionkey@0"
I20260925 05:09:42.452793 3617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 05:09:42.453557 3633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:42.453722 3617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:42.453822 3617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "d8090ca574774263a0c1ba2f8cb1c9a1"
format_stamp: "Formatted at 2026-09-25 05:09:42 on dist-test-slave-4dmq"
server_key: "355e320fb8effc9f1120e3921692c315"
server_key_iv: "a2c74b1fd153879a7de4a2417f67b5ba"
server_key_version: "encryptionkey@0"
I20260925 05:09:42.453917 3617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:42.465870 3617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:42.467533 3617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:42.467698 3617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:42.471520 3617 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:38629
I20260925 05:09:42.471549 3685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:38629 every 8 connection(s)
I20260925 05:09:42.471874 3617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:42.472697 3686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:42.474906 3686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1: Bootstrap starting.
I20260925 05:09:42.475453 3686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:42.475733 3686 log.cc:824] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:42.476397 3686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1: No bootstrap required, opened a new log
I20260925 05:09:42.477859 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3617
I20260925 05:09:42.477978 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:42.478188 14468 external_mini_cluster.cc:1428] Setting key 1f74182592c5d6b53b0ac9b83cb8e93f
I20260925 05:09:42.478338 3686 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38629 } }
I20260925 05:09:42.478471 3686 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:42.478492 3686 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8090ca574774263a0c1ba2f8cb1c9a1, State: Initialized, Role: FOLLOWER
I20260925 05:09:42.478608 3686 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38629 } }
I20260925 05:09:42.478703 3686 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:42.478783 3686 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:42.478854 3686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:42.479978 3686 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38629 } }
I20260925 05:09:42.480082 3686 leader_election.cc:302] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8090ca574774263a0c1ba2f8cb1c9a1; no voters:
I20260925 05:09:42.480221 3686 leader_election.cc:288] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:42.480283 3691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:42.480453 3686 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:42.480468 3691 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [term 1 LEADER]: Becoming Leader. State: Replica: d8090ca574774263a0c1ba2f8cb1c9a1, State: Running, Role: LEADER
I20260925 05:09:42.480561 3691 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38629 } }
I20260925 05:09:42.480914 3691 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8090ca574774263a0c1ba2f8cb1c9a1. Latest consensus state: current_term: 1 leader_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38629 } } }
I20260925 05:09:42.481041 3691 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:42.482086 3692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8090ca574774263a0c1ba2f8cb1c9a1" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 38629 } } }
I20260925 05:09:42.482182 3692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:42.482220 3697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:42.482717 3697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:42.484109 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:42.478656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60445 (local address 127.14.33.62:38629)
0925 05:09:42.478874 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:42.478883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:42.478916 (+ 33us) server_negotiation.cc:407] Connection header received
0925 05:09:42.479035 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:42.479042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:42.479094 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:42.479249 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:42.479406 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:42.480832 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:42.481691 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:42.482329 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:42.482435 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:42.483375 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:42.483380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:42.483381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:42.483627 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:42.483655 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:42.483666 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:42.483763 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:42.483929 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 05:09:42.483955 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":115,"threads_started":1}
I20260925 05:09:42.484583 3697 catalog_manager.cc:1347] Generated new cluster ID: ed347f3e43044eb99b3072582a530c92
I20260925 05:09:42.484673 3697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:42.485075 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:42.492177 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:42.489396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49947 (local address 127.14.33.62:38629)
0925 05:09:42.489446 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:42.489448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:42.489598 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:09:42.489666 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:42.489667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:42.489676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:42.489727 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:42.489864 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:42.490309 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:42.491018 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:42.491216 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:42.491284 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:42.491934 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:42.491935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:42.491935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:42.491939 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:42.491953 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:42.491956 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:42.492027 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:42.492090 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 05:09:42.492103 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 05:09:42.517182 3697 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:42.517688 3697 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:42.524487 3697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8090ca574774263a0c1ba2f8cb1c9a1: Generated new TSK 0
I20260925 05:09:42.524680 3697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:42.528491 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3617
W20260925 05:09:42.532887 3602 connection.cc:535] client connection to 127.14.33.62:38629 recv error: Network error: failed to read from TLS socket (remote: 127.14.33.62:38629): Connection reset by peer (error 104)
2026-09-25T05:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:42 dist-test-slave-4dmq krb5kdc[3717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:42 dist-test-slave-4dmq krb5kdc[3717](info): set up 2 sockets
Sep 25 05:09:42 dist-test-slave-4dmq krb5kdc[3717](info): commencing operation
krb5kdc: starting...
W20260925 05:09:43.554291 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:43 dist-test-slave-4dmq krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:43.667061 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40457
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:43.746380 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:43.746565 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:43.746587 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:43.747612 3730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:43.747645 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:43.747677 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:43.747690 3730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:43.747702 3730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:43.749116 3730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:43.749308 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:43.749457 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:43.754034 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:43.754106 3730 server_base.cc:1034] running on GCE node
W20260925 05:09:43.754072 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:43.754056 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:43.754474 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:43.754796 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:43.755956 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790312983755934 us; error 34 us; skew 500 ppm
Sep 25 05:09:43 dist-test-slave-4dmq krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312983, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:43.758324 3730 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:43.758709 3730 webserver.cc:460] Webserver started at http://127.14.33.62:37359/ using document root <none> and password file <none>
I20260925 05:09:43.758929 3730 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:43.758994 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:43.759135 3730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:43.760118 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "fad5f229610140fb92141e88095345aa"
format_stamp: "Formatted at 2026-09-25 05:09:43 on dist-test-slave-4dmq"
server_key: "b05abb0a083397d08390b6b375e8249d"
server_key_iv: "9c12b86c8d5c1d83ad8470e88bc11e90"
server_key_version: "encryptionkey@0"
I20260925 05:09:43.760469 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "fad5f229610140fb92141e88095345aa"
format_stamp: "Formatted at 2026-09-25 05:09:43 on dist-test-slave-4dmq"
server_key: "b05abb0a083397d08390b6b375e8249d"
server_key_iv: "9c12b86c8d5c1d83ad8470e88bc11e90"
server_key_version: "encryptionkey@0"
I20260925 05:09:43.761863 3730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:43.762537 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:43.762687 3730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:43.762780 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "fad5f229610140fb92141e88095345aa"
format_stamp: "Formatted at 2026-09-25 05:09:43 on dist-test-slave-4dmq"
server_key: "b05abb0a083397d08390b6b375e8249d"
server_key_iv: "9c12b86c8d5c1d83ad8470e88bc11e90"
server_key_version: "encryptionkey@0"
I20260925 05:09:43.762869 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:43.766043 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:43.767669 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:43.767822 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:43.771500 3730 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40457
I20260925 05:09:43.771530 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40457 every 8 connection(s)
I20260925 05:09:43.771826 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:43.772734 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:43.774891 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa: Bootstrap starting.
I20260925 05:09:43.775444 3798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:43.775722 3798 log.cc:824] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:43.776356 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa: No bootstrap required, opened a new log
I20260925 05:09:43.777967 3798 raft_consensus.cc:348] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5f229610140fb92141e88095345aa" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40457 } }
I20260925 05:09:43.778095 3798 raft_consensus.cc:374] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:43.778141 3798 raft_consensus.cc:729] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad5f229610140fb92141e88095345aa, State: Initialized, Role: FOLLOWER
I20260925 05:09:43.778291 3798 consensus_queue.cc:260] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5f229610140fb92141e88095345aa" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40457 } }
I20260925 05:09:43.778362 3798 raft_consensus.cc:388] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:43.778402 3798 raft_consensus.cc:482] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:43.778453 3798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:43.779207 3798 raft_consensus.cc:504] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5f229610140fb92141e88095345aa" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40457 } }
I20260925 05:09:43.779325 3798 leader_election.cc:302] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fad5f229610140fb92141e88095345aa; no voters:
I20260925 05:09:43.779502 3798 leader_election.cc:288] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:43.779549 3801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:43.779737 3801 raft_consensus.cc:686] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [term 1 LEADER]: Becoming Leader. State: Replica: fad5f229610140fb92141e88095345aa, State: Running, Role: LEADER
I20260925 05:09:43.779736 3798 sys_catalog.cc:564] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:43.779913 3801 consensus_queue.cc:237] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5f229610140fb92141e88095345aa" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40457 } }
I20260925 05:09:43.780248 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fad5f229610140fb92141e88095345aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5f229610140fb92141e88095345aa" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40457 } } }
I20260925 05:09:43.780349 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:43.780252 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fad5f229610140fb92141e88095345aa. Latest consensus state: current_term: 1 leader_uuid: "fad5f229610140fb92141e88095345aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5f229610140fb92141e88095345aa" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40457 } } }
I20260925 05:09:43.780481 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:43.780612 3810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:43.781082 3810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:43.781267 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3730
I20260925 05:09:43.781374 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:43.781563 14468 external_mini_cluster.cc:1428] Setting key 9a7091202219bdfaa9ba9c995fc20eb7
I20260925 05:09:43.782745 3810 catalog_manager.cc:1347] Generated new cluster ID: 2fd73ccbc56a4b75992ae76e28ece7c8
I20260925 05:09:43.782799 3810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:43.785909 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:43.782180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50895 (local address 127.14.33.62:40457)
0925 05:09:43.782330 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:43.782339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:43.782368 (+ 29us) server_negotiation.cc:407] Connection header received
0925 05:09:43.782475 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:43.782481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:43.782510 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:43.782651 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:43.785456 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:43.785459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:43.785459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:43.785462 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:43.785487 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:43.785497 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:43.785572 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:43.785658 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 05:09:43.785784 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260925 05:09:43.786862 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:43.798429 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:43.788413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60739 (local address 127.14.33.62:40457)
0925 05:09:43.788447 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:43.788448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:43.797428 (+ 8980us) server_negotiation.cc:407] Connection header received
0925 05:09:43.797502 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:43.797504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:43.797509 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:43.797587 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:43.798192 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:43.798193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:43.798193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:43.798198 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:43.798211 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:43.798216 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:43.798269 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:43.798332 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 05:09:43.798347 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:43.803725 3810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:43.803874 3810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:43.812071 3810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fad5f229610140fb92141e88095345aa: Generated new TSK 0
I20260925 05:09:43.812314 3810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:43.820849 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3730
2026-09-25T05:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:43 dist-test-slave-4dmq krb5kdc[3829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:43 dist-test-slave-4dmq krb5kdc[3829](info): set up 2 sockets
Sep 25 05:09:43 dist-test-slave-4dmq krb5kdc[3829](info): commencing operation
krb5kdc: starting...
W20260925 05:09:44.846526 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:44 dist-test-slave-4dmq krb5kdc[3829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:44.957296 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:43013
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:45.034837 3842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:45.035017 3842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:45.035037 3842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:45.036158 3842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:45.036201 3842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:45.036216 3842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:45.036227 3842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:45.036239 3842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:45.037722 3842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:45.037946 3842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:45.038152 3842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:45.042600 3850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:45.042590 3847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:45.042591 3848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:45.042848 3842 server_base.cc:1034] running on GCE node
I20260925 05:09:45.043066 3842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:45.043318 3842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:45.044502 3842 hybrid_clock.cc:648] HybridClock initialized: now 1790312985044486 us; error 39 us; skew 500 ppm
Sep 25 05:09:45 dist-test-slave-4dmq krb5kdc[3829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:45.046855 3842 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:45.047297 3842 webserver.cc:460] Webserver started at http://127.14.33.62:45823/ using document root <none> and password file <none>
I20260925 05:09:45.047499 3842 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:45.047561 3842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:45.047703 3842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:45.048645 3842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "6a7035a3b36943fd8098ff4d9c7336dc"
format_stamp: "Formatted at 2026-09-25 05:09:45 on dist-test-slave-4dmq"
server_key: "2ffe717a5a7958ec6fb67f649e3d28f9"
server_key_iv: "fc90819213c6ffc37cf918d548e553a4"
server_key_version: "encryptionkey@0"
I20260925 05:09:45.049005 3842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "6a7035a3b36943fd8098ff4d9c7336dc"
format_stamp: "Formatted at 2026-09-25 05:09:45 on dist-test-slave-4dmq"
server_key: "2ffe717a5a7958ec6fb67f649e3d28f9"
server_key_iv: "fc90819213c6ffc37cf918d548e553a4"
server_key_version: "encryptionkey@0"
I20260925 05:09:45.050282 3842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 05:09:45.050971 3857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:45.051128 3842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:45.051229 3842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "6a7035a3b36943fd8098ff4d9c7336dc"
format_stamp: "Formatted at 2026-09-25 05:09:45 on dist-test-slave-4dmq"
server_key: "2ffe717a5a7958ec6fb67f649e3d28f9"
server_key_iv: "fc90819213c6ffc37cf918d548e553a4"
server_key_version: "encryptionkey@0"
I20260925 05:09:45.051299 3842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:45.062395 3842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:45.064198 3842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:45.064352 3842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:45.068343 3842 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:43013
I20260925 05:09:45.068387 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:43013 every 8 connection(s)
I20260925 05:09:45.068707 3842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:45.069561 3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:45.071131 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3842
I20260925 05:09:45.071267 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:45.071453 14468 external_mini_cluster.cc:1428] Setting key 05d45b50705372c6459c554eb41702d3
I20260925 05:09:45.071843 3910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc: Bootstrap starting.
I20260925 05:09:45.072465 3910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:45.072793 3910 log.cc:824] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:45.073575 3910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc: No bootstrap required, opened a new log
I20260925 05:09:45.075577 3910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43013 } }
I20260925 05:09:45.075762 3910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:45.075793 3910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7035a3b36943fd8098ff4d9c7336dc, State: Initialized, Role: FOLLOWER
I20260925 05:09:45.075944 3910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43013 } }
I20260925 05:09:45.076056 3910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:45.076103 3910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:45.076169 3910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:45.076881 3910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43013 } }
I20260925 05:09:45.076999 3910 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7035a3b36943fd8098ff4d9c7336dc; no voters:
I20260925 05:09:45.077186 3910 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:45.077220 3915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:45.077390 3915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [term 1 LEADER]: Becoming Leader. State: Replica: 6a7035a3b36943fd8098ff4d9c7336dc, State: Running, Role: LEADER
I20260925 05:09:45.077443 3910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:45.077527 3915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43013 } }
I20260925 05:09:45.077915 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7035a3b36943fd8098ff4d9c7336dc. Latest consensus state: current_term: 1 leader_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43013 } } }
I20260925 05:09:45.077924 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7035a3b36943fd8098ff4d9c7336dc" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 43013 } } }
I20260925 05:09:45.078009 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:45.078013 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:45.078192 3922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:45.078682 3922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 05:09:45 dist-test-slave-4dmq krb5kdc[3829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:45.082082 3922 catalog_manager.cc:1347] Generated new cluster ID: 59d8d396052742098cbc04d22fcf3c27
I20260925 05:09:45.082127 3922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:45.085809 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:45.071975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39305 (local address 127.14.33.62:43013)
0925 05:09:45.072143 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:45.072151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:45.072175 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:09:45.072607 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:45.072616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:45.072671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:45.072792 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:45.073645 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:45.074542 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:45.075701 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:45.076329 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:45.076410 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:45.081795 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:45.081799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:45.081799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:45.081810 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:45.083828 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:45.084259 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:45.084262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:45.084263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:45.084355 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:45.084570 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:45.084571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:45.084571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:45.084853 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:45.085425 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:45.085492 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 05:09:45.085615 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260925 05:09:45.086789 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:45.096071 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:45.088130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33717 (local address 127.14.33.62:43013)
0925 05:09:45.088161 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:45.088162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:45.089362 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 05:09:45.089392 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:45.089393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:45.089404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:45.089472 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:45.090029 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:45.090777 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:45.091708 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:45.092035 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:45.092108 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:45.093420 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:45.093422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:45.093422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:45.093429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:45.094966 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:45.095325 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:45.095327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:45.095327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:45.095387 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:45.095576 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:45.095577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:45.095577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:45.095706 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:45.095799 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:45.095903 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 05:09:45.095976 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:09:45.100000 3922 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:45.100473 3922 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:45.103861 3922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7035a3b36943fd8098ff4d9c7336dc: Generated new TSK 0
I20260925 05:09:45.104048 3922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:45.106238 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3842
2026-09-25T05:09:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:45 dist-test-slave-4dmq krb5kdc[3941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:45 dist-test-slave-4dmq krb5kdc[3941](info): set up 2 sockets
Sep 25 05:09:45 dist-test-slave-4dmq krb5kdc[3941](info): commencing operation
krb5kdc: starting...
W20260925 05:09:46.130504 14468 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:46 dist-test-slave-4dmq krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:46.241659 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40341
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:46.317119 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:46.317407 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:46.317447 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:46.318567 3954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:46.318630 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:46.318665 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:46.318684 3954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:46.318711 3954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:46.320086 3954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:40341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:46.320338 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:46.320556 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:46.325243 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:46.325301 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:46.325431 3954 server_base.cc:1034] running on GCE node
W20260925 05:09:46.325243 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:46.325738 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:46.326027 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:46.327195 3954 hybrid_clock.cc:648] HybridClock initialized: now 1790312986327178 us; error 28 us; skew 500 ppm
Sep 25 05:09:46 dist-test-slave-4dmq krb5kdc[3941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312986, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:46.329703 3954 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:46.330077 3954 webserver.cc:460] Webserver started at http://127.14.33.62:45343/ using document root <none> and password file <none>
I20260925 05:09:46.330242 3954 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:46.330281 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:46.330384 3954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:46.331660 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "6df2d251d204457fba71cad1c92a517c"
format_stamp: "Formatted at 2026-09-25 05:09:46 on dist-test-slave-4dmq"
server_key: "cc0889d056990067d699c6c8f7dc29fb"
server_key_iv: "4514c128dc0a6cabad74387473016cbc"
server_key_version: "encryptionkey@0"
I20260925 05:09:46.332003 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "6df2d251d204457fba71cad1c92a517c"
format_stamp: "Formatted at 2026-09-25 05:09:46 on dist-test-slave-4dmq"
server_key: "cc0889d056990067d699c6c8f7dc29fb"
server_key_iv: "4514c128dc0a6cabad74387473016cbc"
server_key_version: "encryptionkey@0"
I20260925 05:09:46.333972 3954 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 05:09:46.334906 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:46.335103 3954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 05:09:46.335189 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "6df2d251d204457fba71cad1c92a517c"
format_stamp: "Formatted at 2026-09-25 05:09:46 on dist-test-slave-4dmq"
server_key: "cc0889d056990067d699c6c8f7dc29fb"
server_key_iv: "4514c128dc0a6cabad74387473016cbc"
server_key_version: "encryptionkey@0"
I20260925 05:09:46.335238 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:46.343858 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:46.345568 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:46.345666 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:46.349051 3954 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:40341
I20260925 05:09:46.349097 4021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:40341 every 8 connection(s)
I20260925 05:09:46.349431 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:46.350263 4022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:46.352308 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c: Bootstrap starting.
I20260925 05:09:46.352798 4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:46.353015 4022 log.cc:824] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:46.353755 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c: No bootstrap required, opened a new log
I20260925 05:09:46.355273 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 3954
I20260925 05:09:46.355239 4022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d251d204457fba71cad1c92a517c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40341 } }
I20260925 05:09:46.355413 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:46.355367 4022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:46.355443 4022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df2d251d204457fba71cad1c92a517c, State: Initialized, Role: FOLLOWER
I20260925 05:09:46.355610 14468 external_mini_cluster.cc:1428] Setting key e622a3fa7cb32a4dfcb3ece2ddf603d1
I20260925 05:09:46.355588 4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d251d204457fba71cad1c92a517c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40341 } }
I20260925 05:09:46.355647 4022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:46.355674 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:46.356317 4022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:46.357059 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d251d204457fba71cad1c92a517c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40341 } }
I20260925 05:09:46.357208 4022 leader_election.cc:302] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df2d251d204457fba71cad1c92a517c; no voters:
I20260925 05:09:46.357378 4022 leader_election.cc:288] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:46.357405 4027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:46.357507 4027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [term 1 LEADER]: Becoming Leader. State: Replica: 6df2d251d204457fba71cad1c92a517c, State: Running, Role: LEADER
I20260925 05:09:46.357627 4022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:46.357635 4027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d251d204457fba71cad1c92a517c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40341 } }
I20260925 05:09:46.358003 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df2d251d204457fba71cad1c92a517c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d251d204457fba71cad1c92a517c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40341 } } }
I20260925 05:09:46.358101 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:46.358023 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df2d251d204457fba71cad1c92a517c. Latest consensus state: current_term: 1 leader_uuid: "6df2d251d204457fba71cad1c92a517c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d251d204457fba71cad1c92a517c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 40341 } } }
I20260925 05:09:46.358239 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c [sys.catalog]: This master's current role is: LEADER
W20260925 05:09:46.359457 4043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:46.359510 4043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:46.359561 4033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:46.359687 4033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:46.361194 4033 catalog_manager.cc:1347] Generated new cluster ID: 377429bd405347bcad87e9b3292b7010
I20260925 05:09:46.361250 4033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:46.361500 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:46.356072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41575 (local address 127.14.33.62:40341)
0925 05:09:46.356293 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:46.356301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:46.356331 (+ 30us) server_negotiation.cc:407] Connection header received
0925 05:09:46.356448 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:46.356455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:46.356503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:46.356623 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:46.356987 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:46.358275 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:46.359284 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:46.359851 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:46.359951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:46.360668 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:46.360671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:46.360672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:46.360882 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:46.360919 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:46.360934 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:46.361000 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:46.361134 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 05:09:46.361310 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260925 05:09:46.362425 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:46.366906 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:46.363685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56177 (local address 127.14.33.62:40341)
0925 05:09:46.363732 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:46.363734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:46.363779 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:09:46.363849 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:46.363850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:46.363858 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:46.363904 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:46.364034 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:46.364700 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:46.365783 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:46.366096 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:46.366151 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:46.366710 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:46.366711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:46.366711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:46.366715 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:46.366727 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:46.366731 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:46.366782 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:46.366826 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 05:09:46.366846 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:09:46.383687 4033 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:46.384414 4033 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:46.395908 4033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df2d251d204457fba71cad1c92a517c: Generated new TSK 0
I20260925 05:09:46.396104 4033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:46.405196 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 3954
2026-09-25T05:09:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:46 dist-test-slave-4dmq krb5kdc[4053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:46 dist-test-slave-4dmq krb5kdc[4053](info): set up 2 sockets
Sep 25 05:09:46 dist-test-slave-4dmq krb5kdc[4053](info): commencing operation
krb5kdc: starting...
W20260925 05:09:47.414150 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:47 dist-test-slave-4dmq krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:47.524348 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36759
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:47.613118 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:47.613332 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:47.613353 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:47.614382 4066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:47.614431 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:47.614445 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:47.614459 4066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:47.614470 4066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:47.615934 4066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:47.616118 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:47.616281 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:47.620868 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:47.620857 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:47.620841 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:47.621093 4066 server_base.cc:1034] running on GCE node
I20260925 05:09:47.621347 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:47.621619 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:47.622779 4066 hybrid_clock.cc:648] HybridClock initialized: now 1790312987622763 us; error 47 us; skew 500 ppm
Sep 25 05:09:47 dist-test-slave-4dmq krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312987, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:47.625258 4066 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:47.625653 4066 webserver.cc:460] Webserver started at http://127.14.33.62:36015/ using document root <none> and password file <none>
I20260925 05:09:47.625875 4066 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:47.625941 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:47.626099 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:47.627103 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "6ce42c17c1234db9a47a97e528aed44c"
format_stamp: "Formatted at 2026-09-25 05:09:47 on dist-test-slave-4dmq"
server_key: "2f946e933c7ed11e6f8cf168a9be3c02"
server_key_iv: "ba0cb20adcc805544904c19bc4ad4393"
server_key_version: "encryptionkey@0"
I20260925 05:09:47.627460 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "6ce42c17c1234db9a47a97e528aed44c"
format_stamp: "Formatted at 2026-09-25 05:09:47 on dist-test-slave-4dmq"
server_key: "2f946e933c7ed11e6f8cf168a9be3c02"
server_key_iv: "ba0cb20adcc805544904c19bc4ad4393"
server_key_version: "encryptionkey@0"
I20260925 05:09:47.628717 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:47.629483 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:47.629653 4066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:47.629746 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "6ce42c17c1234db9a47a97e528aed44c"
format_stamp: "Formatted at 2026-09-25 05:09:47 on dist-test-slave-4dmq"
server_key: "2f946e933c7ed11e6f8cf168a9be3c02"
server_key_iv: "ba0cb20adcc805544904c19bc4ad4393"
server_key_version: "encryptionkey@0"
I20260925 05:09:47.629825 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:47.633102 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:47.634841 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:47.634986 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:47.638548 4066 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:36759
I20260925 05:09:47.638594 4133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:36759 every 8 connection(s)
I20260925 05:09:47.638929 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:47.639829 4134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:47.641882 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c: Bootstrap starting.
I20260925 05:09:47.642406 4134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:47.642673 4134 log.cc:824] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:47.643316 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c: No bootstrap required, opened a new log
I20260925 05:09:47.644690 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4066
I20260925 05:09:47.644773 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:47.644966 14468 external_mini_cluster.cc:1428] Setting key 05be44b91654fb3445a6db4283941628
I20260925 05:09:47.644908 4134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce42c17c1234db9a47a97e528aed44c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36759 } }
I20260925 05:09:47.645061 4134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:09:47.645149 4134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce42c17c1234db9a47a97e528aed44c, State: Initialized, Role: FOLLOWER
I20260925 05:09:47.645301 4134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce42c17c1234db9a47a97e528aed44c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36759 } }
I20260925 05:09:47.645396 4134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:09:47.645437 4134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:09:47.645488 4134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:47.646517 4134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce42c17c1234db9a47a97e528aed44c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36759 } }
I20260925 05:09:47.646642 4134 leader_election.cc:302] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ce42c17c1234db9a47a97e528aed44c; no voters:
I20260925 05:09:47.646800 4134 leader_election.cc:288] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:09:47.646974 4140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:47.647053 4134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:47.647058 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:47.645420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49039 (local address 127.14.33.62:36759)
0925 05:09:47.645602 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:47.645618 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:47.645641 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:09:47.645730 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:47.645735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:47.645771 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:47.645879 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:47.646515 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:47.646518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:47.646518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:47.646521 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:47.646554 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:47.646565 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:47.646748 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:47.646813 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 05:09:47.646852 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260925 05:09:47.647104 4140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [term 1 LEADER]: Becoming Leader. State: Replica: 6ce42c17c1234db9a47a97e528aed44c, State: Running, Role: LEADER
I20260925 05:09:47.647171 4140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce42c17c1234db9a47a97e528aed44c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36759 } }
I20260925 05:09:47.647449 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ce42c17c1234db9a47a97e528aed44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce42c17c1234db9a47a97e528aed44c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36759 } } }
I20260925 05:09:47.647516 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:47.647764 4141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ce42c17c1234db9a47a97e528aed44c. Latest consensus state: current_term: 1 leader_uuid: "6ce42c17c1234db9a47a97e528aed44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce42c17c1234db9a47a97e528aed44c" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 36759 } } }
I20260925 05:09:47.647828 4145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:47.647835 4141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:47.648010 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:47.649488 4145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:47.650921 4145 catalog_manager.cc:1347] Generated new cluster ID: 7afa1a09681a441fa86a0964c005e8a0
I20260925 05:09:47.650969 4145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:47.653962 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:47.649293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39727 (local address 127.14.33.62:36759)
0925 05:09:47.649327 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:47.649329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:47.649430 (+ 101us) server_negotiation.cc:407] Connection header received
0925 05:09:47.649494 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:47.649495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:47.649500 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:47.649549 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:47.653741 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:47.653743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:47.653743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:09:47.653747 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:47.653760 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:09:47.653764 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:47.653825 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:47.653893 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:09:47.653905 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 05:09:47.667534 4145 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:47.667678 4145 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:47.669634 4145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ce42c17c1234db9a47a97e528aed44c: Generated new TSK 0
I20260925 05:09:47.669793 4145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:47.673564 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4066
2026-09-25T05:09:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1268 ms)
[----------] 18 tests from AuthTokenIssuingTest (22894 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:47 dist-test-slave-4dmq krb5kdc[4165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:47 dist-test-slave-4dmq krb5kdc[4165](info): set up 2 sockets
Sep 25 05:09:47 dist-test-slave-4dmq krb5kdc[4165](info): commencing operation
krb5kdc: starting...
W20260925 05:09:48.662490 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:48 dist-test-slave-4dmq krb5kdc[4165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:48.771323 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41631
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41631,127.14.33.61:42325,127.14.33.60:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:48.847530 4178 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:09:48.847755 4178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:48.847785 4178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:48.847828 4178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:48.848905 4178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:48.848968 4178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:48.848991 4178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:48.849020 4178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:48.849050 4178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:48.850503 4178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41631,127.14.33.61:42325,127.14.33.60:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:41631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:48.850757 4178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:48.850972 4178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:48.855664 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:48.855664 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:48.855753 4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:48.855983 4178 server_base.cc:1034] running on GCE node
I20260925 05:09:48.856218 4178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:48.856482 4178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:48.857646 4178 hybrid_clock.cc:648] HybridClock initialized: now 1790312988857638 us; error 41 us; skew 500 ppm
Sep 25 05:09:48 dist-test-slave-4dmq krb5kdc[4165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:48.859962 4178 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:48.860394 4178 webserver.cc:460] Webserver started at http://127.14.33.62:46365/ using document root <none> and password file <none>
I20260925 05:09:48.860625 4178 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:48.860683 4178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:48.860846 4178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:48.861935 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "fd10517b31fd434dbde012f9427867cf"
format_stamp: "Formatted at 2026-09-25 05:09:48 on dist-test-slave-4dmq"
server_key: "467befdfd2393fb38e36275788b46acf"
server_key_iv: "a9a6bcb619707bf734e6afcc1f9975a0"
server_key_version: "encryptionkey@0"
I20260925 05:09:48.862296 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "fd10517b31fd434dbde012f9427867cf"
format_stamp: "Formatted at 2026-09-25 05:09:48 on dist-test-slave-4dmq"
server_key: "467befdfd2393fb38e36275788b46acf"
server_key_iv: "a9a6bcb619707bf734e6afcc1f9975a0"
server_key_version: "encryptionkey@0"
I20260925 05:09:48.863583 4178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 05:09:48.864270 4193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:48.864452 4178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:48.864553 4178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "fd10517b31fd434dbde012f9427867cf"
format_stamp: "Formatted at 2026-09-25 05:09:48 on dist-test-slave-4dmq"
server_key: "467befdfd2393fb38e36275788b46acf"
server_key_iv: "a9a6bcb619707bf734e6afcc1f9975a0"
server_key_version: "encryptionkey@0"
I20260925 05:09:48.864622 4178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:48.886441 4178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:48.888094 4178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:48.888252 4178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:48.892071 4178 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:41631
I20260925 05:09:48.892104 4247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:41631 every 8 connection(s)
I20260925 05:09:48.892453 4178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:48.893424 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:48.894795 4248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:48.895255 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4178
I20260925 05:09:48.895383 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:48.895584 14468 external_mini_cluster.cc:1428] Setting key 6c51c5f5f8131599a41c0d7da29e40e5
Sep 25 05:09:48 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.61@KRBTEST.COM; defaulting to no policy
I20260925 05:09:48.905442 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:48.895534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41631 (local address 127.0.0.1:42854)
0925 05:09:48.895743 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:48.895788 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:48.896491 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:48.896728 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:48.896730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:48.897081 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:48.897295 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:48.897300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:48.898828 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:48.898829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:48.899599 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:48.899602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:48.900311 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:48.900312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:48.900365 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:48.900894 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:48.900902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:48.902242 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:48.904072 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:48.904078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:48.904080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:48.904261 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:48.904524 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:48.904537 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:48.904537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:48.904634 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:48.904931 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:48.904935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:48.904982 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 05:09:48.905046 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:09:48.905208 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260925 05:09:48.905447 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:48.895574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.14.33.62:41631)
0925 05:09:48.895719 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:48.895727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:48.895883 (+ 156us) server_negotiation.cc:407] Connection header received
0925 05:09:48.896616 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:48.896618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:48.896633 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:48.896703 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:48.897382 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:48.898708 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:48.899684 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:48.900203 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:48.900311 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:48.902326 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:48.902334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:48.902334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:48.902342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:48.904004 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:48.904328 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:48.904330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:48.904330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:48.904399 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:48.904695 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:48.904695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:48.904695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:48.904876 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:48.904932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:48.905085 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 05:09:48.905293 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.61@KRBTEST.COM" created.
I20260925 05:09:48.906442 4248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:48.906769 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:48.906674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40092)
0925 05:09:48.906707 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:48.906750 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 05:09:48.906930 4197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
I20260925 05:09:48.907331 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:48.907252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40102)
0925 05:09:48.907293 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:48.907316 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 05:09:48.907434 4248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:42325: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab.
Entry for principal kudu/127.14.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab.
I20260925 05:09:48.919957 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:42325
--webserver_interface=127.14.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab
--principal=kudu/127.14.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41631,127.14.33.61:42325,127.14.33.60:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:09:48.956728 4248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } attempt: 1
I20260925 05:09:48.957311 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:48.957173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40114)
0925 05:09:48.957241 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:48.957283 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:09:48.957665 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:48.957584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40116)
0925 05:09:48.957626 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:48.957650 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:09:48.957772 4248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:42325: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
W20260925 05:09:48.997426 4255 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:09:48.997604 4255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:48.997622 4255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:48.997640 4255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:48.998658 4255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:48.998692 4255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:48.998704 4255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:48.998718 4255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:48.998729 4255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:49.000079 4255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41631,127.14.33.61:42325,127.14.33.60:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab
--principal=kudu/127.14.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:49.000293 4255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:49.000439 4255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:49.004612 4260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:49.004647 4263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:49.004623 4261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:49.004899 4255 server_base.cc:1034] running on GCE node
I20260925 05:09:49.005112 4255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:49.005420 4255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:49.006568 4255 hybrid_clock.cc:648] HybridClock initialized: now 1790312989006559 us; error 29 us; skew 500 ppm
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:49.008735 4255 init.cc:377] Logged in from keytab as kudu/127.14.33.61@KRBTEST.COM (short username kudu)
I20260925 05:09:49.009179 4255 webserver.cc:460] Webserver started at http://127.14.33.61:38013/ using document root <none> and password file <none>
I20260925 05:09:49.009400 4255 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:49.009473 4255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:49.009621 4255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:49.010555 4255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data/instance:
uuid: "da9548791b2b444e9dbacce9c07dbf0d"
format_stamp: "Formatted at 2026-09-25 05:09:49 on dist-test-slave-4dmq"
server_key: "83820d5fa7e4ed16f8f41f4b5475206d"
server_key_iv: "461e3d34553d687c6444f079280cd5a0"
server_key_version: "encryptionkey@0"
I20260925 05:09:49.010892 4255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal/instance:
uuid: "da9548791b2b444e9dbacce9c07dbf0d"
format_stamp: "Formatted at 2026-09-25 05:09:49 on dist-test-slave-4dmq"
server_key: "83820d5fa7e4ed16f8f41f4b5475206d"
server_key_iv: "461e3d34553d687c6444f079280cd5a0"
server_key_version: "encryptionkey@0"
I20260925 05:09:49.012136 4255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:49.012795 4270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:49.012941 4255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:49.013027 4255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal
uuid: "da9548791b2b444e9dbacce9c07dbf0d"
format_stamp: "Formatted at 2026-09-25 05:09:49 on dist-test-slave-4dmq"
server_key: "83820d5fa7e4ed16f8f41f4b5475206d"
server_key_iv: "461e3d34553d687c6444f079280cd5a0"
server_key_version: "encryptionkey@0"
I20260925 05:09:49.013103 4255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:49.025898 4248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } attempt: 2
I20260925 05:09:49.026295 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.026144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40120)
0925 05:09:49.026212 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.026259 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:09:49.026618 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.026542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40134)
0925 05:09:49.026578 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.026601 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 05:09:49.026724 4248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:42325: Network error: Client connection negotiation failed: client connection to 127.14.33.61:42325: connect: Connection refused (error 111)
I20260925 05:09:49.041514 4255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:49.043182 4255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:49.043327 4255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:49.047013 4255 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.61:42325
I20260925 05:09:49.047080 4323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.61:42325 every 8 connection(s)
I20260925 05:09:49.047382 4255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
I20260925 05:09:49.048305 4324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:49.049803 4324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:49.054234 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4255
I20260925 05:09:49.054342 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-1/wal/instance
I20260925 05:09:49.054530 14468 external_mini_cluster.cc:1428] Setting key a9a827758dcec73cd2de35617e5f0a47
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:49.058866 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.050379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41631 (local address 127.0.0.1:42860)
0925 05:09:49.050546 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.050578 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.050990 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.051188 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.051191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.051411 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.051582 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.051585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.052198 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.052199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.052971 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.052973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.053378 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.053379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.053479 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.053946 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.053954 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.055788 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.057481 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.057487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.057488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.057722 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.057963 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.057964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.057964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.058067 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.058332 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.058339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.058464 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.058591 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.058725 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260925 05:09:49.058873 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.050420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.14.33.62:41631)
0925 05:09:49.050498 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.050500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.050650 (+ 150us) server_negotiation.cc:407] Connection header received
0925 05:09:49.051065 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.051066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.051075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.051124 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.051650 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.052098 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.053040 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.053299 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.053351 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.055900 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.055901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.055902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.055910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.057414 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.057830 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.057831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.057832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.057895 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.058146 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.058147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.058147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.058268 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.058340 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.058647 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.058738 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:09:49.059499 4324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.33.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.60@KRBTEST.COM" created.
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:49.069018 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.059737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40146)
0925 05:09:49.059790 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.059802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.059847 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.060046 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.060047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.060201 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.060260 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.060262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.061093 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.061094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.061914 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.061916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.062343 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.062344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.062427 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.062752 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.062757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.066008 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.067938 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.067939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.067940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.068095 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.068323 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.068323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.068324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.068381 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.068645 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.068646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.068697 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.068784 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.068880 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 05:09:49.069105 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.059772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.14.33.61:42325)
0925 05:09:49.059906 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.059910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.059927 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:09:49.059949 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.059955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.059966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.060022 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.060335 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.061003 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.061987 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.062264 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.062316 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.066111 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.066118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.066118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.066123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.067869 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.068195 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.068198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.068198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.068257 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.068444 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.068445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.068446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.068576 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.068665 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.068828 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.068950 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260925 05:09:49.069624 4324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:49.069959 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.069845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43402)
0925 05:09:49.069892 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.069932 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:09:49.070086 4272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
I20260925 05:09:49.070502 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.070412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43406)
0925 05:09:49.070459 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.070482 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 05:09:49.070603 4324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:44003: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab.
Entry for principal kudu/127.14.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab.
I20260925 05:09:49.079730 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:44003
--webserver_interface=127.14.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab
--principal=kudu/127.14.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:38477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:41631,127.14.33.61:42325,127.14.33.60:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:09:49.117871 4248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } attempt: 3
I20260925 05:09:49.119810 4324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } attempt: 1
I20260925 05:09:49.120159 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.120046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43418)
0925 05:09:49.120099 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.120133 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 05:09:49.120534 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.120435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43420)
0925 05:09:49.120485 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.120513 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 05:09:49.120611 4324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:44003: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:49.125224 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.118330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.14.33.61:42325)
0925 05:09:49.118391 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.118393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.118410 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:09:49.118434 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.118435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.118445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.118506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.118860 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.119335 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.120316 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.120615 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.120731 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.122764 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.122766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.122766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.122774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.124227 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.124538 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.124539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.124539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.124597 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.124774 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.124775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.124775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.124897 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.124960 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.125061 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.125147 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 05:09:49.125232 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.118267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40150)
0925 05:09:49.118329 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.118340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.118381 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.118533 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.118534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.118709 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.118788 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.118791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.119413 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.119415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.120225 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.120227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.120761 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.120762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.120851 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.121318 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.121324 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.122680 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.124314 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.124315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.124316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.124474 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.124659 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.124660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.124660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.124716 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.124955 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.124956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.124992 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.125040 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.125109 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 05:09:49.125654 4248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:49.125960 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.125868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43436)
0925 05:09:49.125911 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.125935 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 05:09:49.126560 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.126486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43452)
0925 05:09:49.126525 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.126546 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:09:49.126680 4248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:44003: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
I20260925 05:09:49.157853 4248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } attempt: 1
I20260925 05:09:49.158285 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.158150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43466)
0925 05:09:49.158213 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.158253 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:09:49.158674 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.158556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43468)
0925 05:09:49.158618 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.158648 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:09:49.158843 4248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:44003: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
W20260925 05:09:49.160387 4331 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:09:49.160616 4331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:49.160650 4331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:49.160691 4331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:49.161821 4331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:49.161880 4331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:49.161904 4331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:49.161929 4331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:49.161958 4331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:49.163430 4331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:38477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:41631,127.14.33.61:42325,127.14.33.60:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab
--principal=kudu/127.14.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:49.163689 4331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:49.163909 4331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:09:49.168486 4331 server_base.cc:1034] running on GCE node
W20260925 05:09:49.168562 4336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:49.168558 4337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:49.168658 4339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:49.168984 4331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:49.169314 4331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:49.170470 4331 hybrid_clock.cc:648] HybridClock initialized: now 1790312989170451 us; error 35 us; skew 500 ppm
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:49.172823 4331 init.cc:377] Logged in from keytab as kudu/127.14.33.60@KRBTEST.COM (short username kudu)
I20260925 05:09:49.173381 4331 webserver.cc:460] Webserver started at http://127.14.33.60:44627/ using document root <none> and password file <none>
I20260925 05:09:49.173620 4331 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:49.173686 4331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:49.173827 4331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:49.174777 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data/instance:
uuid: "ccc745dc55b84b3aba1546eb5f535b08"
format_stamp: "Formatted at 2026-09-25 05:09:49 on dist-test-slave-4dmq"
server_key: "7734eef13a8359cf38278775cfe1afc1"
server_key_iv: "46d85e6da7111370528dbf3bc8e72d70"
server_key_version: "encryptionkey@0"
I20260925 05:09:49.175123 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal/instance:
uuid: "ccc745dc55b84b3aba1546eb5f535b08"
format_stamp: "Formatted at 2026-09-25 05:09:49 on dist-test-slave-4dmq"
server_key: "7734eef13a8359cf38278775cfe1afc1"
server_key_iv: "46d85e6da7111370528dbf3bc8e72d70"
server_key_version: "encryptionkey@0"
I20260925 05:09:49.176333 4331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 05:09:49.177006 4346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:49.177194 4331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:49.177299 4331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal
uuid: "ccc745dc55b84b3aba1546eb5f535b08"
format_stamp: "Formatted at 2026-09-25 05:09:49 on dist-test-slave-4dmq"
server_key: "7734eef13a8359cf38278775cfe1afc1"
server_key_iv: "46d85e6da7111370528dbf3bc8e72d70"
server_key_version: "encryptionkey@0"
I20260925 05:09:49.177366 4331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:49.188740 4324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } attempt: 2
I20260925 05:09:49.189153 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.188996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43480)
0925 05:09:49.189053 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.189087 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 05:09:49.189494 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.189393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43490)
0925 05:09:49.189440 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.189464 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 05:09:49.189580 4324 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:44003: Network error: Client connection negotiation failed: client connection to 127.14.33.60:44003: connect: Connection refused (error 111)
I20260925 05:09:49.195583 4331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:49.197340 4331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:49.197484 4331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:49.201164 4331 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.60:44003
I20260925 05:09:49.201208 4399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.60:44003 every 8 connection(s)
I20260925 05:09:49.201630 4331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
I20260925 05:09:49.202508 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:49.204011 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4331
I20260925 05:09:49.204105 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312729583056-14468-0/minicluster-data/master-2/wal/instance
I20260925 05:09:49.203943 4400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:49.204288 14468 external_mini_cluster.cc:1428] Setting key 5d1ec4db10a973e5120dad5fe5cb85eb
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:49.213502 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.204752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.14.33.62:41631)
0925 05:09:49.204804 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.204806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.205162 (+ 356us) server_negotiation.cc:407] Connection header received
0925 05:09:49.205788 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.205789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.205802 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.205861 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.206356 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.206868 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.207901 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.208197 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.208275 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.210565 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.210566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.210566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.210573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.212135 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.212504 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.212505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.212506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.212564 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.212822 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.212823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.212823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.212953 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.213026 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.213325 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.213402 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 05:09:49.213631 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.204690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:41631 (local address 127.0.0.1:42876)
0925 05:09:49.205029 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.205060 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.205644 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.205893 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.205895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.206124 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.206268 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.206272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.206940 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.206941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.207824 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.207826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.208323 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.208324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.208442 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.208964 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.208972 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.210467 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.212199 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.212205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.212207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.212412 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.212644 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.212645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.212645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.212748 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.213022 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.213026 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.213187 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.213280 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.213388 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":83,"threads_started":1}
I20260925 05:09:49.214190 4400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:49.214990 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.204721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.14.33.60:44003)
0925 05:09:49.205215 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.205219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.205246 (+ 27us) server_negotiation.cc:407] Connection header received
0925 05:09:49.205649 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.205657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.205683 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.205773 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.206412 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.207428 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.208292 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.208707 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.208760 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.211143 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.211150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.211151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.211157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.213346 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.213790 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.213792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.213793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.213934 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.214154 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.214155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.214156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.214332 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.214462 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.214748 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.214886 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":79,"threads_started":1}
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:49.221088 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.214441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:42325 (local address 127.0.0.1:40158)
0925 05:09:49.214480 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.214490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.214526 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.214669 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.214669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.214875 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.214918 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.214919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.215706 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.215707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.216515 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.216517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.216980 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.216981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.217044 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.217464 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.217469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.218744 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.220125 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.220125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.220126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.220338 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.220508 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.220509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.220509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.220569 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.220823 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.220824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.220865 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.220948 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.221027 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 05:09:49.221192 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.214500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.14.33.61:42325)
0925 05:09:49.214549 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.214551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.214560 (+ 9us) server_negotiation.cc:407] Connection header received
0925 05:09:49.214585 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.214586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.214598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.214668 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.214972 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.215634 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.216586 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.216895 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.216956 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.218840 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.218842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.218842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.218848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.220052 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.220407 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.220408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.220408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.220454 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.220661 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.220661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.220661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.220755 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.220805 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.220985 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.221056 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 05:09:49.221524 4400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:49.224148 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.215696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.14.33.62:41631)
0925 05:09:49.215729 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.215731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.215738 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:09:49.215797 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.215797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.215804 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.215840 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.216378 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.216997 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.218220 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.218518 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.218579 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.220998 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.220999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.220999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.221005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.222916 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.223336 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.223337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.223338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.223417 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.223644 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.223645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.223645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.223756 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.223940 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.223977 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.224052 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:49.227972 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.221735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43504)
0925 05:09:49.221782 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.221791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.221824 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.221973 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.221973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.222135 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.222165 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.222166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.223081 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.223082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.223548 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.223549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.223885 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.223885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.223929 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.224341 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.224346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.225431 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.227043 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.227044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.227044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.227256 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.227452 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.227452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.227453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.227499 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.227718 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.227718 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.227750 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.227802 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.227881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 05:09:49.228014 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.221790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.14.33.60:44003)
0925 05:09:49.221822 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.221824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.221831 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:09:49.221877 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.221877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.221887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.221947 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.222253 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.222995 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.223627 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.223819 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.223887 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.225509 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.225510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.225510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.225516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.226949 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.227322 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.227323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.227323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.227368 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.227570 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.227570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.227571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.227661 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.227723 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.227828 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.227920 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":17}
I20260925 05:09:49.229566 4400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: Bootstrap starting.
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:49.230479 4400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:49.230787 4400 log.cc:824] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:49.231587 4400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: No bootstrap required, opened a new log
I20260925 05:09:49.232762 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.225196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.14.33.61:42325)
0925 05:09:49.225265 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.225267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.225275 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:49.225294 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.225295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.225303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.225382 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.225919 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.226604 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.227359 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.227559 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.227632 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.230218 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.230219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.230219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.230226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.231729 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.232047 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.232049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.232049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.232103 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.232298 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.232299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.232299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.232395 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.232565 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.232599 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.232675 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:09:49.233197 4400 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.233315 4400 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc745dc55b84b3aba1546eb5f535b08, State: Initialized, Role: FOLLOWER
I20260925 05:09:49.233395 4400 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.233721 4400 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:49.233984 4248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } attempt: 2
I20260925 05:09:49.234565 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:49.235172 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.235265 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:49.235555 4221 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd10517b31fd434dbde012f9427867cf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42886
W20260925 05:09:49.235667 4417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:49.236510 4417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312988, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:49.237005 4331 master_runner.cc:186] Error getting master registration for 127.14.33.62:41631: OK, instance_id { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" instance_seqno: 1790312988891001 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:09:49.239348 4331 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:09:49.241390 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.234236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.14.33.60:44003)
0925 05:09:49.234381 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.234382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.234389 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:09:49.234406 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.234406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.234415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.234467 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.234738 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.235384 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.236319 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.236607 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.236671 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.238604 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.238605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.238605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.238611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.240183 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.240581 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.240582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.240582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.240645 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.240850 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.240850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.240851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.240992 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.241063 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.241248 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.241315 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 05:09:49.241403 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.234206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43508)
0925 05:09:49.234272 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.234282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.234318 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.234502 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.234503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.234644 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.234678 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.234680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.235469 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.235470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.236187 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.236190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.236717 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.236718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.236767 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.237148 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.237153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.238493 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.240269 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.240270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.240271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.240502 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.240718 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.240719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.240719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.240783 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.241063 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.241063 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.241145 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.241214 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.241305 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 05:09:49.242779 4248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf: Bootstrap starting.
I20260925 05:09:49.243439 4248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:49.243732 4248 log.cc:824] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:49.244508 4248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf: No bootstrap required, opened a new log
I20260925 05:09:49.245821 4248 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.245954 4248 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd10517b31fd434dbde012f9427867cf, State: Initialized, Role: FOLLOWER
I20260925 05:09:49.246027 4248 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.246287 4248 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:49.246331 4221 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:09:49.246258 4420 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.246398 4221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:49.247186 4221 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.247671 4373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd10517b31fd434dbde012f9427867cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ccc745dc55b84b3aba1546eb5f535b08"
I20260925 05:09:49.247804 4373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [term 0 FOLLOWER]: Advancing to term 1
W20260925 05:09:49.248522 4431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:49.248569 4431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:49.248760 4373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd10517b31fd434dbde012f9427867cf in term 1.
I20260925 05:09:49.248912 4221 leader_election.cc:288] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [CANDIDATE]: Term 1 election: Requested vote from peers da9548791b2b444e9dbacce9c07dbf0d (127.14.33.61:42325), ccc745dc55b84b3aba1546eb5f535b08 (127.14.33.60:44003)
I20260925 05:09:49.249053 4195 leader_election.cc:302] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccc745dc55b84b3aba1546eb5f535b08, fd10517b31fd434dbde012f9427867cf; no voters:
I20260925 05:09:49.249279 4433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:49.249400 4433 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [term 1 LEADER]: Becoming Leader. State: Replica: fd10517b31fd434dbde012f9427867cf, State: Running, Role: LEADER
I20260925 05:09:49.249482 4433 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.249862 4433 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd10517b31fd434dbde012f9427867cf. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.249926 4433 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:49.250087 4440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:49.250192 4420 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:49.250262 4440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:49.249033 4178 master_runner.cc:186] Error getting master registration for 127.14.33.61:42325: OK, instance_id { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" instance_seqno: 1790312989046014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:09:49.251524 4178 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:09:49.252804 4373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [term 1 FOLLOWER]: Refusing update from remote peer fd10517b31fd434dbde012f9427867cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:49.253360 4420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:49.254526 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd10517b31fd434dbde012f9427867cf. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.254616 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:49.260022 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.250941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.14.33.60:44003)
0925 05:09:49.250974 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.250976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.252123 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 05:09:49.252210 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.252211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.252235 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.252262 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.253060 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.253730 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.254682 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.254958 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.254994 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.256676 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.256677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.256678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.256685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.258364 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.258953 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.258954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.258954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.259023 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.259314 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.259315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.259316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.259454 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.259536 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.259855 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.259929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:49.260459 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.250718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.14.33.62:41631)
0925 05:09:49.250759 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.250761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.250929 (+ 168us) server_negotiation.cc:407] Connection header received
0925 05:09:49.251019 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.251020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.251029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.251056 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.251915 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.252546 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.255199 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.255495 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.255555 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.257201 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.257202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.257202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.257209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.258961 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.259324 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.259325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.259325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.259373 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.259558 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.259559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.259560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.259672 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.259768 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.260299 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.260364 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 05:09:49.261770 4420 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.261861 4420 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:49.261971 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.262046 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:49.262183 4445 mvcc.cc:205] Tried to move back new op lower bound from 7333122003977244672 to 7333122003966644224. Current Snapshot: MvccSnapshot[applied={T|T < 7333122003977244672}]
I20260925 05:09:49.263257 4444 mvcc.cc:205] Tried to move back new op lower bound from 7333122003977244672 to 7333122003966644224. Current Snapshot: MvccSnapshot[applied={T|T < 7333122003977244672}]
I20260925 05:09:49.267856 4440 catalog_manager.cc:1347] Generated new cluster ID: 5d09cfae92c54acdabd4622cbf8d4083
I20260925 05:09:49.267907 4440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:49.268252 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.251097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.14.33.61:42325)
0925 05:09:49.251129 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.251130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.260088 (+ 8958us) server_negotiation.cc:407] Connection header received
0925 05:09:49.260861 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.260863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.260874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.260923 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.261534 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.262216 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.263236 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.263539 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.263632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.265017 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.265018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.265019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.265026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.266744 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.267131 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.267133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.267133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.267194 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.267504 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.267506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.267506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.267627 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.267728 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.268076 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.268172 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:09:49.280709 4324 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } attempt: 3
I20260925 05:09:49.281730 4440 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:49.282698 4440 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 05:09:49 dist-test-slave-4dmq krb5kdc[4165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312989, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:49.287147 4440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf: Generated new TSK 0
I20260925 05:09:49.287323 4440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:09:49.288956 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.281046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.14.33.60:44003)
0925 05:09:49.281093 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.281094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.281102 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:49.281137 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.281138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.281148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.281198 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.281472 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.282159 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.283081 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.283388 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.283449 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.285589 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:49.285590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:49.285591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:49.285597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:49.287504 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.288244 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.288245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.288245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.288297 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:49.288471 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:49.288472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:49.288472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:49.288595 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:49.288651 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:49.288807 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 05:09:49.288868 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:09:49.288957 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:49.280942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:44003 (local address 127.0.0.1:43520)
0925 05:09:49.281020 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:49.281029 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:49.281063 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:49.281212 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:49.281213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:49.281371 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:49.281413 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.281415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.282231 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.282232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.282998 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:49.283000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:49.283501 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:49.283502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:49.283558 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:49.283999 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:49.284013 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:49.285474 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:49.287936 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.287936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.287937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.288162 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.288345 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:49.288346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:49.288347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:49.288413 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:49.288652 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:49.288654 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:49.288710 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 05:09:49.288774 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 05:09:49.288866 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 05:09:49.290586 4324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: Bootstrap starting.
I20260925 05:09:49.291350 4324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:49.291646 4324 log.cc:824] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:49.292613 4324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: No bootstrap required, opened a new log
I20260925 05:09:49.294811 4324 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.294950 4324 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9548791b2b444e9dbacce9c07dbf0d, State: Initialized, Role: FOLLOWER
I20260925 05:09:49.295027 4324 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } }
I20260925 05:09:49.295354 4324 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:49.296871 4298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd10517b31fd434dbde012f9427867cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "da9548791b2b444e9dbacce9c07dbf0d"
I20260925 05:09:49.297277 4298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:49.297935 4298 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd10517b31fd434dbde012f9427867cf in term 1.
I20260925 05:09:49.299312 4297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [term 1 FOLLOWER]: Refusing update from remote peer fd10517b31fd434dbde012f9427867cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:49.299580 4433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:49.302011 4463 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.302116 4463 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:49.302175 4463 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd10517b31fd434dbde012f9427867cf. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.302222 4463 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:49.302254 4463 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.302311 4463 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:09:49.309695 4462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:49.309746 4462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:49.310292 4464 mvcc.cc:205] Tried to move back new op lower bound from 7333122004114952192 to 7333122003966644224. Current Snapshot: MvccSnapshot[applied={T|T < 7333122003977244672}]
W20260925 05:09:49.312090 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.310514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.14.33.61:42325)
0925 05:09:49.310550 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.310552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.311462 (+ 910us) server_negotiation.cc:407] Connection header received
0925 05:09:49.311567 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.311568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.311579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.311613 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.312033 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40192: BlockingRecv error: recv got EOF from 127.0.0.1:40192 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 05:09:49.312129 4251 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.310896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.14.33.62:41631)
0925 05:09:49.310943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.310945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.311642 (+ 697us) server_negotiation.cc:407] Connection header received
0925 05:09:49.311770 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.311772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.311785 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.311813 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.312072 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42900: BlockingRecv error: recv got EOF from 127.0.0.1:42900 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 05:09:49.312848 4404 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.311199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.14.33.60:44003)
0925 05:09:49.311232 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.311234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.312362 (+ 1128us) server_negotiation.cc:407] Connection header received
0925 05:09:49.312459 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.312460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.312472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.312501 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.312792 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43532: BlockingRecv error: recv got EOF from 127.0.0.1:43532 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 05:09:49.314846 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.314231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.14.33.61:42325)
0925 05:09:49.314271 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.314273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.314388 (+ 115us) server_negotiation.cc:407] Connection header received
0925 05:09:49.314479 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.314480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.314492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.314518 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.314810 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40202: BlockingRecv error: recv got EOF from 127.0.0.1:40202 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 05:09:49.317863 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.317013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.14.33.61:42325)
0925 05:09:49.317045 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.317047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.317360 (+ 313us) server_negotiation.cc:407] Connection header received
0925 05:09:49.317470 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.317471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.317480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.317507 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.317824 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40206: BlockingRecv error: recv got EOF from 127.0.0.1:40206 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 05:09:49.321560 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.320811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.14.33.61:42325)
0925 05:09:49.320841 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.320843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.321058 (+ 215us) server_negotiation.cc:407] Connection header received
0925 05:09:49.321196 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.321197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.321207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.321233 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.321525 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40220: BlockingRecv error: recv got EOF from 127.0.0.1:40220 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 05:09:49.327226 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.326481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.14.33.61:42325)
0925 05:09:49.326513 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.326526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.326751 (+ 225us) server_negotiation.cc:407] Connection header received
0925 05:09:49.326840 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.326841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.326850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.326877 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.327186 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40232: BlockingRecv error: recv got EOF from 127.0.0.1:40232 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 05:09:49.336784 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.336085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.14.33.61:42325)
0925 05:09:49.336113 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.336115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.336351 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:09:49.336437 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.336438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.336447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.336473 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.336745 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40236: BlockingRecv error: recv got EOF from 127.0.0.1:40236 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 05:09:49.354599 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.353823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.14.33.61:42325)
0925 05:09:49.353865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.353867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.354103 (+ 236us) server_negotiation.cc:407] Connection header received
0925 05:09:49.354205 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.354206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.354215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.354241 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.354552 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40244: BlockingRecv error: recv got EOF from 127.0.0.1:40244 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 05:09:49.388710 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.387792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.14.33.61:42325)
0925 05:09:49.387834 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.387836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.388131 (+ 295us) server_negotiation.cc:407] Connection header received
0925 05:09:49.388252 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.388253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.388264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.388293 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.388655 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40260: BlockingRecv error: recv got EOF from 127.0.0.1:40260 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 05:09:49.454799 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.454096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.14.33.61:42325)
0925 05:09:49.454172 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.454175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.454286 (+ 111us) server_negotiation.cc:407] Connection header received
0925 05:09:49.454338 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.454339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.454350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.454410 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.454733 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40276: BlockingRecv error: recv got EOF from 127.0.0.1:40276 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 05:09:49.584708 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.583922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.14.33.61:42325)
0925 05:09:49.584016 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.584018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.584116 (+ 98us) server_negotiation.cc:407] Connection header received
0925 05:09:49.584223 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.584225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.584236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.584307 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.584642 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40278: BlockingRecv error: recv got EOF from 127.0.0.1:40278 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 05:09:49.842535 4328 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:49.841845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.14.33.61:42325)
0925 05:09:49.841917 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:49.841919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:49.841929 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:09:49.842024 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:49.842025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:49.842038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:49.842109 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:49.842470 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40286: BlockingRecv error: recv got EOF from 127.0.0.1:40286 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:09:49.860745 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd10517b31fd434dbde012f9427867cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd10517b31fd434dbde012f9427867cf" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 41631 } } peers { permanent_uuid: "da9548791b2b444e9dbacce9c07dbf0d" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 42325 } } peers { permanent_uuid: "ccc745dc55b84b3aba1546eb5f535b08" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 44003 } } }
I20260925 05:09:49.860893 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd10517b31fd434dbde012f9427867cf [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:50.236975 4417 catalog_manager.cc:1259] Loaded cluster ID: 5d09cfae92c54acdabd4622cbf8d4083
I20260925 05:09:50.237051 4417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: loading cluster ID for follower catalog manager: success
I20260925 05:09:50.237823 4417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: acquiring CA information for follower catalog manager: success
I20260925 05:09:50.238054 4417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccc745dc55b84b3aba1546eb5f535b08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:09:50.310139 4462 catalog_manager.cc:1259] Loaded cluster ID: 5d09cfae92c54acdabd4622cbf8d4083
I20260925 05:09:50.310206 4462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: loading cluster ID for follower catalog manager: success
I20260925 05:09:50.311012 4462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: acquiring CA information for follower catalog manager: success
I20260925 05:09:50.311271 4462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da9548791b2b444e9dbacce9c07dbf0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:09:50.358340 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:50.355616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.14.33.61:42325)
0925 05:09:50.355729 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:50.355732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:50.355810 (+ 78us) server_negotiation.cc:407] Connection header received
0925 05:09:50.355909 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:50.355910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:50.355982 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:50.356066 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:50.356218 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.356706 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.357516 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.357729 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.357804 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:50.357981 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:50.358112 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:50.358197 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:50.358219 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 05:09:50.358248 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260925 05:09:50.362450 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:50.360050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.14.33.60:44003)
0925 05:09:50.360188 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:50.360190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:50.360308 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:09:50.360358 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:50.360359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:50.360395 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:50.360467 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:50.360549 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.361032 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.361803 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.361990 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.362044 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:50.362171 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:50.362271 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:50.362324 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:50.362359 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 05:09:50.362374 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 05:09:50.366720 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:50.364048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.14.33.61:42325)
0925 05:09:50.364086 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:50.364088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:50.364182 (+ 94us) server_negotiation.cc:407] Connection header received
0925 05:09:50.364244 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:50.364245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:50.364257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:50.364323 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:50.364444 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.365196 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.365990 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.366245 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.366304 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:50.366473 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:50.366503 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:50.366554 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:50.366630 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:09:50.366653 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:50.367175 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:50.364354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.14.33.60:44003)
0925 05:09:50.364398 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:50.364400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:50.364543 (+ 143us) server_negotiation.cc:407] Connection header received
0925 05:09:50.364586 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:50.364587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:50.364596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:50.364646 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:50.364759 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.365430 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.366350 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.366644 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.366714 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:50.366886 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:50.366911 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:50.366950 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:50.367076 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 05:09:50.367095 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:50.367357 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:50.364034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.14.33.62:41631)
0925 05:09:50.364134 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:50.364137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:50.364246 (+ 109us) server_negotiation.cc:407] Connection header received
0925 05:09:50.364365 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:50.364366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:50.364418 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:50.364476 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:50.364681 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.365245 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.366251 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:50.366554 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:50.366589 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:50.366918 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:50.367003 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:50.367045 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:50.367264 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 05:09:50.367288 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
I20260925 05:09:50.368942 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4178
I20260925 05:09:50.373999 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4255
I20260925 05:09:50.378634 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4331
2026-09-25T05:09:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2706 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:09:50 dist-test-slave-4dmq krb5kdc[4555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:09:50 dist-test-slave-4dmq krb5kdc[4555](info): set up 2 sockets
Sep 25 05:09:50 dist-test-slave-4dmq krb5kdc[4555](info): commencing operation
krb5kdc: starting...
W20260925 05:09:51.374473 14468 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/test-user.keytab.
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:09:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.33.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
Entry for principal kudu/127.14.33.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab.
I20260925 05:09:51.483307 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:37347
--webserver_interface=127.14.33.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:37347,127.14.33.61:41975,127.14.33.60:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:09:51.559640 4568 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:09:51.559880 4568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:51.559909 4568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:51.559950 4568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:51.561055 4568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:51.561120 4568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:51.561161 4568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:51.561189 4568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:51.561213 4568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:51.562599 4568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:37347,127.14.33.61:41975,127.14.33.60:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.62.keytab
--principal=kudu/127.14.33.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.62:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.33.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:51.562877 4568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:51.563089 4568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:51.567813 4574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:51.567886 4576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:51.567978 4568 server_base.cc:1034] running on GCE node
W20260925 05:09:51.567839 4573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:51.568320 4568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:51.568642 4568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:51.569818 4568 hybrid_clock.cc:648] HybridClock initialized: now 1790312991569814 us; error 37 us; skew 500 ppm
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:51.572113 4568 init.cc:377] Logged in from keytab as kudu/127.14.33.62@KRBTEST.COM (short username kudu)
I20260925 05:09:51.572516 4568 webserver.cc:460] Webserver started at http://127.14.33.62:44483/ using document root <none> and password file <none>
I20260925 05:09:51.572729 4568 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:51.572796 4568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:51.572943 4568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:51.573994 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data/instance:
uuid: "78058eedf35a4714969b77e07e97062e"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "7971c3ec6b1c5af083b64f55b919c51e"
server_key_iv: "45eed473dda8352e9d8894b56c3b7e24"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.574352 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal/instance:
uuid: "78058eedf35a4714969b77e07e97062e"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "7971c3ec6b1c5af083b64f55b919c51e"
server_key_iv: "45eed473dda8352e9d8894b56c3b7e24"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.575592 4568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:51.576267 4583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:51.576435 4568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:51.576532 4568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal
uuid: "78058eedf35a4714969b77e07e97062e"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "7971c3ec6b1c5af083b64f55b919c51e"
server_key_iv: "45eed473dda8352e9d8894b56c3b7e24"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.576615 4568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:51.587838 4568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:51.589579 4568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:51.589748 4568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:51.593489 4568 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.62:37347
I20260925 05:09:51.593518 4635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.62:37347 every 8 connection(s)
I20260925 05:09:51.593843 4568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/data/info.pb
I20260925 05:09:51.594789 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:51.596264 4636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.597452 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4568
I20260925 05:09:51.597527 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-0/wal/instance
I20260925 05:09:51.597747 14468 external_mini_cluster.cc:1428] Setting key 535be9c6413670daa99c657f9333ef34
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.61@KRBTEST.COM; defaulting to no policy
I20260925 05:09:51.606842 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.596882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:37347 (local address 127.0.0.1:52358)
0925 05:09:51.597034 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.597084 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.597635 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.597929 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.597931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.598161 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.598313 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.598317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.599647 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.599649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.600709 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.600712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.601273 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.601274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.601360 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.601844 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.601854 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.603406 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.605228 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.605233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.605235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.605420 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.605667 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.605668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.605668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.605759 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.606244 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.606249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.606299 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.606406 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.606651 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":99,"thread_start_us":85,"threads_started":1}
I20260925 05:09:51.606844 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.596932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.14.33.62:37347)
0925 05:09:51.597085 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.597094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.597207 (+ 113us) server_negotiation.cc:407] Connection header received
0925 05:09:51.597783 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.597786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.597808 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.597922 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.598426 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.599556 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.600802 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.601100 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.601269 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.603492 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.603499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.603499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.603506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.605073 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.605514 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.605517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.605517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.605594 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.605827 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.605828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.605828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.606043 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.606160 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.606465 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.606590 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.61@KRBTEST.COM" created.
I20260925 05:09:51.607653 4636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.607980 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.607871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46806)
0925 05:09:51.607922 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.607961 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 05:09:51.608105 4585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
I20260925 05:09:51.608533 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.608442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46822)
0925 05:09:51.608487 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.608510 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:09:51.608671 4636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:41975: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab.
Entry for principal kudu/127.14.33.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab.
I20260925 05:09:51.621594 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:41975
--webserver_interface=127.14.33.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab
--principal=kudu/127.14.33.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:37347,127.14.33.61:41975,127.14.33.60:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:09:51.657886 4636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } attempt: 1
I20260925 05:09:51.658311 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.658186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46834)
0925 05:09:51.658247 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.658285 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:09:51.658638 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.658547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46846)
0925 05:09:51.658590 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.658613 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 05:09:51.658712 4636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:41975: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
W20260925 05:09:51.699410 4643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:09:51.699644 4643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:51.699692 4643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:51.699726 4643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:51.700798 4643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:51.700881 4643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:51.700905 4643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:51.700930 4643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:51.700958 4643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:51.702397 4643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:37347,127.14.33.61:41975,127.14.33.60:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.61.keytab
--principal=kudu/127.14.33.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.61:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.33.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:51.702675 4643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:51.702898 4643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:51.707438 4648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:51.707440 4651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:51.707484 4649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:51.707620 4643 server_base.cc:1034] running on GCE node
I20260925 05:09:51.707862 4643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:51.708150 4643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:51.709311 4643 hybrid_clock.cc:648] HybridClock initialized: now 1790312991709295 us; error 33 us; skew 500 ppm
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:51.711592 4643 init.cc:377] Logged in from keytab as kudu/127.14.33.61@KRBTEST.COM (short username kudu)
I20260925 05:09:51.712026 4643 webserver.cc:460] Webserver started at http://127.14.33.61:38667/ using document root <none> and password file <none>
I20260925 05:09:51.712242 4643 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:51.712319 4643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:51.712476 4643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:51.713588 4643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data/instance:
uuid: "ddca9906bbc84aaf916bf534ecfbf0a0"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "0dd8717837ef0a1f5e03afc43aa540b6"
server_key_iv: "751eec6bc16daa2cd329780ee3b42a5c"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.713950 4643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal/instance:
uuid: "ddca9906bbc84aaf916bf534ecfbf0a0"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "0dd8717837ef0a1f5e03afc43aa540b6"
server_key_iv: "751eec6bc16daa2cd329780ee3b42a5c"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.715178 4643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 05:09:51.715904 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:51.716068 4643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 05:09:51.716162 4643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal
uuid: "ddca9906bbc84aaf916bf534ecfbf0a0"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "0dd8717837ef0a1f5e03afc43aa540b6"
server_key_iv: "751eec6bc16daa2cd329780ee3b42a5c"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.716245 4643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:51.726868 4636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } attempt: 2
I20260925 05:09:51.727253 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.727126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46852)
0925 05:09:51.727191 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.727225 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 05:09:51.727543 4643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:51.727571 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.727478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46854)
0925 05:09:51.727521 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.727544 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 05:09:51.727674 4636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.61:41975: Network error: Client connection negotiation failed: client connection to 127.14.33.61:41975: connect: Connection refused (error 111)
I20260925 05:09:51.729291 4643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:51.729439 4643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:51.732887 4643 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.61:41975
I20260925 05:09:51.732928 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.61:41975 every 8 connection(s)
I20260925 05:09:51.733274 4643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/data/info.pb
I20260925 05:09:51.734266 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:51.735702 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4643
I20260925 05:09:51.735827 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-1/wal/instance
I20260925 05:09:51.735682 4712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.736012 14468 external_mini_cluster.cc:1428] Setting key 27f25b521dc52035742985ee108f6a9c
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.33.60@KRBTEST.COM; defaulting to no policy
I20260925 05:09:51.744589 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.736400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.14.33.62:37347)
0925 05:09:51.736457 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.736459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.736667 (+ 208us) server_negotiation.cc:407] Connection header received
0925 05:09:51.737157 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.737158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.737170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.737215 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.737768 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.738238 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.739157 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.739372 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.739448 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.741814 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.741815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.741816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.741824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.743307 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.743649 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.743649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.743650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.743698 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.743973 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.743975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.743975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.744101 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.744173 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.744427 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.744509 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 05:09:51.744815 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.736378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:37347 (local address 127.0.0.1:52372)
0925 05:09:51.736533 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.736563 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.737040 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.737277 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.737280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.737497 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.737674 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.737677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.738328 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.738329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.739066 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.739067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.739477 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.739478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.739587 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.740062 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.740076 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.741649 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.743388 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.743394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.743396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.743586 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.743803 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.743806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.743806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.743901 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.744166 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.744172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.744285 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.744377 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.744528 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.33.60@KRBTEST.COM" created.
I20260925 05:09:51.745324 4712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:51.752820 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.745567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46856)
0925 05:09:51.745618 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.745629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.745671 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.745908 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.745909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.746060 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.746114 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.746115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.747325 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.747327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.748138 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.748140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.748569 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.748570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.748646 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.748987 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.748992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.750212 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.751988 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.751988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.751989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.752125 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.752311 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.752312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.752312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.752355 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.752620 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.752621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.752648 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.752697 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.752752 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 05:09:51.752912 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.745599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.14.33.61:41975)
0925 05:09:51.745750 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.745753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.745770 (+ 17us) server_negotiation.cc:407] Connection header received
0925 05:09:51.745807 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.745809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.745823 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.745888 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.746185 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.747222 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.748211 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.748497 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.748546 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.750293 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.750300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.750301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.750308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.751902 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.752194 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.752196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.752196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.752251 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.752407 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.752408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.752408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.752558 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.752635 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.752736 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.752825 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":104,"threads_started":1}
I20260925 05:09:51.753454 4712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.753759 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.753642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56712)
0925 05:09:51.753696 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.753734 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 05:09:51.753908 4660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
I20260925 05:09:51.754262 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.754180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56724)
0925 05:09:51.754223 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.754244 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 05:09:51.754356 4712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:39621: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.33.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab.
Entry for principal kudu/127.14.33.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab.
I20260925 05:09:51.759370 14468 external_mini_cluster.cc:1326] Running /tmp/dist-test-task85DeA_/build/release/bin/kudu
/tmp/dist-test-task85DeA_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:39621
--webserver_interface=127.14.33.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab
--principal=kudu/127.14.33.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.33.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.33.62:37347,127.14.33.61:41975,127.14.33.60:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:09:51.803565 4712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } attempt: 1
I20260925 05:09:51.804064 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.803902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56732)
0925 05:09:51.803967 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.804014 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 05:09:51.804541 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.804411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56736)
0925 05:09:51.804484 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.804517 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 05:09:51.804677 4712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:39621: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
I20260925 05:09:51.818806 4636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } attempt: 3
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:51.826296 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.819166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46866)
0925 05:09:51.819215 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.819233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.819307 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.819509 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.819510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.819715 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.819782 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.819784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.820475 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.820476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.821028 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.821030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.821525 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.821526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.821631 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.822113 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.822119 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.823338 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.825196 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.825197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.825197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.825383 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.825630 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.825631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.825631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.825696 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.825953 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.825953 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.825999 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.826100 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.826197 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 05:09:51.826292 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.819218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.14.33.61:41975)
0925 05:09:51.819281 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.819283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.819295 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:09:51.819401 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.819402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.819414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.819495 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.819876 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.820371 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.821147 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.821398 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.821523 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.823444 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.823445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.823446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.823454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.825013 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.825486 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.825487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.825488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.825550 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.825762 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.825763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.825763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.825880 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.825960 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.826134 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.826205 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 05:09:51.826644 4636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.827019 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.826891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56746)
0925 05:09:51.826952 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.826990 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 05:09:51.827374 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.827256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56748)
0925 05:09:51.827311 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.827351 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 05:09:51.827482 4636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:39621: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
W20260925 05:09:51.835455 4719 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:09:51.835625 4719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:09:51.835641 4719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:09:51.835660 4719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:09:51.836675 4719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:09:51.836707 4719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:09:51.836720 4719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:09:51.836735 4719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:09:51.836746 4719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:09:51.838148 4719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.33.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.33.62:37347,127.14.33.61:41975,127.14.33.60:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/krb5kdc/kudu_127.14.33.60.keytab
--principal=kudu/127.14.33.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.33.60:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.33.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 04:15:06 UTC on ccycloud.quasar-mlmkqt.root.comops.site
build id 4671826
I20260925 05:09:51.838321 4719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:09:51.838460 4719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:09:51.842793 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:51.842816 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:09:51.842796 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:09:51.843043 4719 server_base.cc:1034] running on GCE node
I20260925 05:09:51.843248 4719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:09:51.843549 4719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:09:51.844688 4719 hybrid_clock.cc:648] HybridClock initialized: now 1790312991844666 us; error 34 us; skew 500 ppm
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:09:51.846848 4719 init.cc:377] Logged in from keytab as kudu/127.14.33.60@KRBTEST.COM (short username kudu)
I20260925 05:09:51.847213 4719 webserver.cc:460] Webserver started at http://127.14.33.60:41647/ using document root <none> and password file <none>
I20260925 05:09:51.847407 4719 fs_manager.cc:359] Metadata directory not provided
I20260925 05:09:51.847445 4719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:09:51.847569 4719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:09:51.848446 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data/instance:
uuid: "6d820f0fdc694a85a42a8c3aca6b194e"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "b6215fb7d0214637f1b2a907cd0d71f2"
server_key_iv: "1a28fc1cd39a8799396305038fa1cbb9"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.848771 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal/instance:
uuid: "6d820f0fdc694a85a42a8c3aca6b194e"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "b6215fb7d0214637f1b2a907cd0d71f2"
server_key_iv: "1a28fc1cd39a8799396305038fa1cbb9"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.850019 4719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 05:09:51.850659 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:09:51.850864 4719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 05:09:51.850966 4719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data,/tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal
uuid: "6d820f0fdc694a85a42a8c3aca6b194e"
format_stamp: "Formatted at 2026-09-25 05:09:51 on dist-test-slave-4dmq"
server_key: "b6215fb7d0214637f1b2a907cd0d71f2"
server_key_iv: "1a28fc1cd39a8799396305038fa1cbb9"
server_key_version: "encryptionkey@0"
I20260925 05:09:51.851035 4719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:09:51.858641 4636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } attempt: 1
I20260925 05:09:51.859016 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.858917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56756)
0925 05:09:51.858962 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.858992 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 05:09:51.859310 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.859242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56770)
0925 05:09:51.859277 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.859297 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 05:09:51.859411 4636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.33.60:39621: Network error: Client connection negotiation failed: client connection to 127.14.33.60:39621: connect: Connection refused (error 111)
I20260925 05:09:51.863054 4719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:09:51.865226 4719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:09:51.865362 4719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:09:51.868745 4719 rpc_server.cc:307] RPC server started. Bound to: 127.14.33.60:39621
I20260925 05:09:51.868790 4786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.33.60:39621 every 8 connection(s)
I20260925 05:09:51.869109 4719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/data/info.pb
I20260925 05:09:51.870090 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:09:51.871374 4787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.872819 4712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } attempt: 2
I20260925 05:09:51.873263 14468 external_mini_cluster.cc:1388] Started /tmp/dist-test-task85DeA_/build/release/bin/kudu as pid 4719
I20260925 05:09:51.873365 14468 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task85DeA_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312729583056-14468-0/minicluster-data/master-2/wal/instance
I20260925 05:09:51.873569 14468 external_mini_cluster.cc:1428] Setting key 9c0b759dfa0b6c1ddb98832de7275bd8
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.61@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:51.881346 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.871983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.14.33.62:37347)
0925 05:09:51.872026 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.872028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.872218 (+ 190us) server_negotiation.cc:407] Connection header received
0925 05:09:51.872624 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.872625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.872633 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.872676 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.873416 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.874145 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.875398 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.875582 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.875659 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.878155 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.878156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.878157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.878165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.879768 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.880307 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.880308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.880308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.880383 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.880641 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.880642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.880642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.880759 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.880828 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.881161 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.881241 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:51.881691 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.871951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.62:37347 (local address 127.0.0.1:52384)
0925 05:09:51.872100 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.872137 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.872541 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.872728 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.872731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.873010 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.873291 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.873297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.874219 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.874220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.875318 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.875320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.875688 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.875689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.875802 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.876439 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.876449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.878067 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.879860 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.879868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.879869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.880200 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.880461 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.880462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.880462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.880563 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.880833 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.880840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.880977 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.881100 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.881330 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":171,"thread_start_us":85,"threads_started":1}
I20260925 05:09:51.882146 4787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.882458 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.873050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56772)
0925 05:09:51.873109 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.873142 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.873181 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.873574 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.873576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.873770 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.873809 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.873811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.874925 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.874926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.875638 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.875641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.876170 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.876171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.876229 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.876604 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.876608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.879065 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.881440 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.881441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.881442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.881638 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.881846 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.881847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.881847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.881906 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.882173 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.882175 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.882214 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.882277 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.882369 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 05:09:51.882467 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.873248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.14.33.60:39621)
0925 05:09:51.873416 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.873420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.873438 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:09:51.873457 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.873458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.873475 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.873547 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.873924 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.874849 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.875714 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.876101 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.876177 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.879138 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.879145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.879146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.879151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.881346 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.881726 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.881727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.881727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.881777 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.881974 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.881975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.881975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.882119 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.882181 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.882314 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.882377 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260925 05:09:51.883234 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.873916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.14.33.60:39621)
0925 05:09:51.874046 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.874048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.874129 (+ 81us) server_negotiation.cc:407] Connection header received
0925 05:09:51.874200 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.874200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.874210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.874279 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.874979 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.875608 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.876500 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.876781 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.876844 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.879264 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.879265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.879265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.879270 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.881377 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.881717 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.881719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.881720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.881929 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.882280 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.882281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.882281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.882398 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.882669 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.883088 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.883151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260925 05:09:51.884778 4712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: Bootstrap starting.
I20260925 05:09:51.885699 4712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:51.886019 4712 log.cc:824] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: Log is configured to *not* fsync() on all Append() calls
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:51.887333 4712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: No bootstrap required, opened a new log
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.62@KRBTEST.COM
I20260925 05:09:51.888562 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.882371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.61:41975 (local address 127.0.0.1:46870)
0925 05:09:51.882483 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.882490 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.882521 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.882638 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.882639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.882774 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.882803 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.882804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.883563 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.883564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.884265 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.884267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.884623 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.884624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.884679 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.885022 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.885027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.886343 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.887597 (+ 1254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.887597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.887598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.887750 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.887933 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.887933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.887934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.887992 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.888290 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.888291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.888333 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.888404 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.888481 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 05:09:51.888592 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.882455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.14.33.61:41975)
0925 05:09:51.882548 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.882549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.882556 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:09:51.882573 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.882574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.882582 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.882634 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.882861 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.883491 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.884352 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.884546 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.884622 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.886407 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.886408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.886408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.886414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.887517 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.887829 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.887830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.887830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.887877 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.888060 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.888060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.888061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.888215 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.888278 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.888442 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.888515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 05:09:51.888947 4787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:09:51.889510 4712 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.889645 4712 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddca9906bbc84aaf916bf534ecfbf0a0, State: Initialized, Role: FOLLOWER
I20260925 05:09:51.889734 4712 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.890027 4712 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:51.892093 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
W20260925 05:09:51.892143 4805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:51.892271 4805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:51.892365 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.60@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:51.893261 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.883862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.14.33.62:37347)
0925 05:09:51.883896 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.883898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.883942 (+ 44us) server_negotiation.cc:407] Connection header received
0925 05:09:51.883996 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.883997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.884005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.884058 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.884593 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.885208 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.886090 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.886353 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.886407 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.888999 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.889000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.889000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.889007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.890572 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.890983 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.890985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.890985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.891044 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.891286 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.891287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.891287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.891415 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.891557 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.891796 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.893194 (+ 1398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:51.892345 4643 master_runner.cc:186] Error getting master registration for 127.14.33.62:37347: OK, instance_id { permanent_uuid: "78058eedf35a4714969b77e07e97062e" instance_seqno: 1790312991592358 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:09:51.895272 4643 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:09:51.896099 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.889150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56786)
0925 05:09:51.889222 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.889231 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.889270 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.889615 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.889616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.889740 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.889767 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.889768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.890461 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.890463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.891042 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.891045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.891492 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.891493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.891535 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.891943 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.891948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.893252 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.894967 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.894968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.894968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.895127 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.895365 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.895366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.895366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.895438 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.895746 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.895747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.895791 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.895873 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.895966 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 05:09:51.896174 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.889452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.14.33.60:39621)
0925 05:09:51.889507 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.889509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.889516 (+ 7us) server_negotiation.cc:407] Connection header received
0925 05:09:51.889533 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.889533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.889541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.889588 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.889857 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.890373 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.891125 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.891414 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.891667 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.893355 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.893356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.893357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.893364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.894888 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.895203 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.895204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.895204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.895272 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.895525 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.895526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.895526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.895669 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.895736 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.895922 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.896024 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":37}
I20260925 05:09:51.897338 4787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: Bootstrap starting.
I20260925 05:09:51.897962 4787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:51.898273 4787 log.cc:824] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:51.899034 4787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: No bootstrap required, opened a new log
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.33.61@KRBTEST.COM
I20260925 05:09:51.900988 4787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.901194 4787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d820f0fdc694a85a42a8c3aca6b194e, State: Initialized, Role: FOLLOWER
I20260925 05:09:51.901297 4787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.901587 4787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:51.901561 4808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.901757 4808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:51.903229 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.895373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.14.33.61:41975)
0925 05:09:51.895427 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.895429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.895437 (+ 8us) server_negotiation.cc:407] Connection header received
0925 05:09:51.895455 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.895456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.895465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.895524 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.896085 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.896694 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.897584 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.897749 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.897805 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.900259 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.900260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.900261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.900268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.901848 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.902218 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.902219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.902219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.902276 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.902472 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.902473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.902473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.902577 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.902719 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.903079 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.903140 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 05:09:51.903756 4819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:51.903853 4819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:51.904392 14468 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:09:51.903576 4719 master_runner.cc:186] Error getting master registration for 127.14.33.62:37347: OK, instance_id { permanent_uuid: "78058eedf35a4714969b77e07e97062e" instance_seqno: 1790312991592358 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:09:51.905755 4719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:09:51.905498 4610 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "78058eedf35a4714969b77e07e97062e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52390
I20260925 05:09:51.934542 4636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } attempt: 2
Sep 25 05:09:51 dist-test-slave-4dmq krb5kdc[4555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.33.62@KRBTEST.COM for kudu/127.14.33.60@KRBTEST.COM
I20260925 05:09:51.942631 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.934974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.14.33.60:39621)
0925 05:09:51.935013 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.935015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.935026 (+ 11us) server_negotiation.cc:407] Connection header received
0925 05:09:51.935072 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.935073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.935082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.935129 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.935462 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.936188 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.937201 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.937499 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.937560 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.939809 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.939811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.939811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.939819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.941424 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.941785 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.941786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.941786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.941836 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.942008 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.942009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.942009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.942121 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.942173 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.942480 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.942547 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 05:09:51.942905 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.934842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.33.60:39621 (local address 127.0.0.1:56790)
0925 05:09:51.934953 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 05:09:51.934963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 05:09:51.935002 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:09:51.935150 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:09:51.935151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:09:51.935331 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:09:51.935383 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.935385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.936269 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.936270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.937071 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:09:51.937073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.937615 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.937617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:09:51.937678 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.938177 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:09:51.938182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:09:51.939713 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:09:51.941511 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.941512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.941512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.941710 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.941898 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:09:51.941898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:09:51.941899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:09:51.941950 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:09:51.942197 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:09:51.942198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:09:51.942252 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 05:09:51.942311 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 05:09:51.942397 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 05:09:51.944397 4636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e: Bootstrap starting.
I20260925 05:09:51.945147 4636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e: Neither blocks nor log segments found. Creating new log.
I20260925 05:09:51.945493 4636 log.cc:824] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:09:51.946414 4636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e: No bootstrap required, opened a new log
I20260925 05:09:51.947698 4636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.947839 4636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78058eedf35a4714969b77e07e97062e, State: Initialized, Role: FOLLOWER
I20260925 05:09:51.947913 4636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.948179 4636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:09:51.948276 4610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:09:51.948371 4610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:51.949062 4610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.949290 4610 leader_election.cc:288] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [CANDIDATE]: Term 1 election: Requested vote from peers ddca9906bbc84aaf916bf534ecfbf0a0 (127.14.33.61:41975), 6d820f0fdc694a85a42a8c3aca6b194e (127.14.33.60:39621)
W20260925 05:09:51.950117 4833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:09:51.950168 4833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:09:51.950728 4686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78058eedf35a4714969b77e07e97062e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0"
I20260925 05:09:51.950858 4686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:51.952018 4686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78058eedf35a4714969b77e07e97062e in term 1.
I20260925 05:09:51.952230 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78058eedf35a4714969b77e07e97062e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d820f0fdc694a85a42a8c3aca6b194e"
I20260925 05:09:51.952337 4761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:09:51.953440 4585 leader_election.cc:302] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78058eedf35a4714969b77e07e97062e, ddca9906bbc84aaf916bf534ecfbf0a0; no voters:
I20260925 05:09:51.954481 4761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78058eedf35a4714969b77e07e97062e in term 1.
I20260925 05:09:51.948184 4822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.955360 4822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:51.955451 4822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:09:51.955541 4822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [term 1 LEADER]: Becoming Leader. State: Replica: 78058eedf35a4714969b77e07e97062e, State: Running, Role: LEADER
I20260925 05:09:51.955889 4568 master_runner.cc:263] No leader master found from master 78058eedf35a4714969b77e07e97062e
I20260925 05:09:51.955935 4568 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:09:51.955844 4822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } }
I20260925 05:09:51.956607 4822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78058eedf35a4714969b77e07e97062e. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.957700 4822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:51.965358 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.955756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.14.33.61:41975)
0925 05:09:51.955792 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.955794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.957987 (+ 2193us) server_negotiation.cc:407] Connection header received
0925 05:09:51.958077 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.958078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.958087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.958114 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.958870 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.959568 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.960570 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.960843 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.960879 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.962317 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.962318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.962319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.962325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.964003 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.964391 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.964392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.964393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.964459 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.964769 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.964770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.964770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.964889 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.964957 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.965172 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.965250 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 05:09:51.971498 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.956111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.14.33.62:37347)
0925 05:09:51.956160 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.956162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.965242 (+ 9080us) server_negotiation.cc:407] Connection header received
0925 05:09:51.965335 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.965336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.965344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.965371 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.966087 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.966510 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.967075 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.967316 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.967375 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.968676 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.968677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.968678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.968685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.970332 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.970722 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.970724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.970724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.970776 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.970964 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.970965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.970966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.971081 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.971145 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.971341 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.971411 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 05:09:51.972002 4844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:09:51.972175 4844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:09:51.976030 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:51.956382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.14.33.60:39621)
0925 05:09:51.956414 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:51.956416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:51.961282 (+ 4866us) server_negotiation.cc:407] Connection header received
0925 05:09:51.961368 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:51.961369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:51.961377 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:51.961403 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:51.969541 (+ 8138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.970305 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.971309 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:51.971601 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:51.971638 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:51.973271 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:09:51.973273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:09:51.973273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:09:51.973280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:09:51.974892 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.975260 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.975261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.975261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.975311 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:09:51.975503 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:09:51.975504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:09:51.975504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:09:51.975615 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:09:51.975698 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:51.975879 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 05:09:51.975945 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 05:09:51.978034 4761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [term 1 FOLLOWER]: Refusing update from remote peer 78058eedf35a4714969b77e07e97062e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:51.978035 4686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [term 1 FOLLOWER]: Refusing update from remote peer 78058eedf35a4714969b77e07e97062e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:09:51.978312 4822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:51.978557 4822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:09:51.979460 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78058eedf35a4714969b77e07e97062e. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.979545 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 05:09:51.980605 4664 tablet.cc:2369] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:09:51.980577 4808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78058eedf35a4714969b77e07e97062e. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.980656 4808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:51.989979 4822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.990058 4822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:51.990693 4845 mvcc.cc:205] Tried to move back new op lower bound from 7333122015139545088 to 7333122015052918784. Current Snapshot: MvccSnapshot[applied={T|T < 7333122015139545088}]
I20260925 05:09:51.991115 4848 mvcc.cc:205] Tried to move back new op lower bound from 7333122015139545088 to 7333122015052918784. Current Snapshot: MvccSnapshot[applied={T|T < 7333122015139545088}]
I20260925 05:09:51.992633 4808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.992722 4808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:51.993784 4839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.993878 4839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e [sys.catalog]: This master's current role is: LEADER
I20260925 05:09:51.994252 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78058eedf35a4714969b77e07e97062e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78058eedf35a4714969b77e07e97062e" member_type: VOTER last_known_addr { host: "127.14.33.62" port: 37347 } } peers { permanent_uuid: "ddca9906bbc84aaf916bf534ecfbf0a0" member_type: VOTER last_known_addr { host: "127.14.33.61" port: 41975 } } peers { permanent_uuid: "6d820f0fdc694a85a42a8c3aca6b194e" member_type: VOTER last_known_addr { host: "127.14.33.60" port: 39621 } } }
I20260925 05:09:51.994328 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:09:51.994885 4844 catalog_manager.cc:1347] Generated new cluster ID: e0d46068856c4f508f5145858420487b
I20260925 05:09:51.994933 4844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:09:52.008109 4844 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:09:52.008906 4844 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:09:52.023483 4844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78058eedf35a4714969b77e07e97062e: Generated new TSK 0
I20260925 05:09:52.023669 4844 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:09:52.044386 4639 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.042740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.14.33.62:37347)
0925 05:09:52.042785 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.042787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.043744 (+ 957us) server_negotiation.cc:407] Connection header received
0925 05:09:52.043854 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.043855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.043867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.043898 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.044324 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52396: BlockingRecv error: recv got EOF from 127.0.0.1:52396 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:09:52.045135 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.043129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.14.33.61:41975)
0925 05:09:52.043166 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.043167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.044614 (+ 1447us) server_negotiation.cc:407] Connection header received
0925 05:09:52.044710 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.044711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.044735 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.044765 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.045060 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46900: BlockingRecv error: recv got EOF from 127.0.0.1:46900 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 05:09:52.045831 4792 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.043450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.14.33.60:39621)
0925 05:09:52.043487 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.043489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.045347 (+ 1858us) server_negotiation.cc:407] Connection header received
0925 05:09:52.045441 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.045442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.045452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.045480 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.045776 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56810: BlockingRecv error: recv got EOF from 127.0.0.1:56810 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 05:09:52.047631 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.046944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.14.33.61:41975)
0925 05:09:52.046974 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.046976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.047136 (+ 160us) server_negotiation.cc:407] Connection header received
0925 05:09:52.047230 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.047231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.047246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.047274 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.047592 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46910: BlockingRecv error: recv got EOF from 127.0.0.1:46910 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 05:09:52.054153 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.053384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.14.33.61:41975)
0925 05:09:52.053416 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.053418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.053655 (+ 237us) server_negotiation.cc:407] Connection header received
0925 05:09:52.053762 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.053763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.053773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.053801 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.054094 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46920: BlockingRecv error: recv got EOF from 127.0.0.1:46920 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 05:09:52.057928 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.057232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.14.33.61:41975)
0925 05:09:52.057263 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.057265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.057460 (+ 195us) server_negotiation.cc:407] Connection header received
0925 05:09:52.057549 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.057550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.057559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.057586 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.057891 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46936: BlockingRecv error: recv got EOF from 127.0.0.1:46936 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 05:09:52.063761 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.062984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.14.33.61:41975)
0925 05:09:52.063026 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.063028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.063260 (+ 232us) server_negotiation.cc:407] Connection header received
0925 05:09:52.063380 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.063381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.063391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.063419 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.063711 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46944: BlockingRecv error: recv got EOF from 127.0.0.1:46944 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 05:09:52.073660 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.072869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.14.33.61:41975)
0925 05:09:52.072901 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.072902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.073107 (+ 205us) server_negotiation.cc:407] Connection header received
0925 05:09:52.073288 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.073289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.073299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.073327 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.073620 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46958: BlockingRecv error: recv got EOF from 127.0.0.1:46958 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 05:09:52.091588 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.090892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.14.33.61:41975)
0925 05:09:52.090933 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.090935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.091107 (+ 172us) server_negotiation.cc:407] Connection header received
0925 05:09:52.091179 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.091180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.091189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.091256 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.091543 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46962: BlockingRecv error: recv got EOF from 127.0.0.1:46962 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 05:09:52.125211 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.124608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.14.33.61:41975)
0925 05:09:52.124671 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.124673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.124741 (+ 68us) server_negotiation.cc:407] Connection header received
0925 05:09:52.124800 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.124801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.124811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.124852 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.125135 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46972: BlockingRecv error: recv got EOF from 127.0.0.1:46972 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 05:09:52.191740 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.190901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.14.33.61:41975)
0925 05:09:52.190987 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.190990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.191100 (+ 110us) server_negotiation.cc:407] Connection header received
0925 05:09:52.191171 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.191172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.191185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.191261 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.191642 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46980: BlockingRecv error: recv got EOF from 127.0.0.1:46980 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 05:09:52.321604 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.320863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.14.33.61:41975)
0925 05:09:52.320952 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.320954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.321012 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:09:52.321062 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.321063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.321075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.321208 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.321539 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46984: BlockingRecv error: recv got EOF from 127.0.0.1:46984 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 05:09:52.579600 4716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:09:52.578952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.14.33.61:41975)
0925 05:09:52.579025 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:52.579028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:52.579038 (+ 10us) server_negotiation.cc:407] Connection header received
0925 05:09:52.579099 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:52.579100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:52.579112 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:52.579183 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:09:52.579540 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46990: BlockingRecv error: recv got EOF from 127.0.0.1:46990 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 05:09:52.892689 4805 catalog_manager.cc:1259] Loaded cluster ID: e0d46068856c4f508f5145858420487b
I20260925 05:09:52.892764 4805 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: loading cluster ID for follower catalog manager: success
I20260925 05:09:52.893533 4805 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: acquiring CA information for follower catalog manager: success
I20260925 05:09:52.893834 4805 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddca9906bbc84aaf916bf534ecfbf0a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:09:52.904384 4819 catalog_manager.cc:1259] Loaded cluster ID: e0d46068856c4f508f5145858420487b
I20260925 05:09:52.904430 4819 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: loading cluster ID for follower catalog manager: success
I20260925 05:09:52.905104 4819 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: acquiring CA information for follower catalog manager: success
I20260925 05:09:52.905392 4819 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d820f0fdc694a85a42a8c3aca6b194e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:09:53.094961 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:53.092659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.14.33.61:41975)
0925 05:09:53.092782 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:53.092784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:53.092799 (+ 15us) server_negotiation.cc:407] Connection header received
0925 05:09:53.092845 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:53.092846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:53.092896 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:53.092964 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:53.093105 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.093561 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.094238 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.094424 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.094470 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:53.094607 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:53.094710 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:53.094771 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:53.094816 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 05:09:53.094832 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260925 05:09:53.099017 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:53.096734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.14.33.60:39621)
0925 05:09:53.096828 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:53.096830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:53.096852 (+ 22us) server_negotiation.cc:407] Connection header received
0925 05:09:53.096895 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:53.096895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:53.096927 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:53.096987 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:53.097070 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.097590 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.098263 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.098451 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.098498 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:53.098666 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:53.098762 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:53.098833 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:53.098869 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 05:09:53.098887 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
I20260925 05:09:53.103268 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:53.100606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.14.33.61:41975)
0925 05:09:53.100652 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:53.100653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:53.100761 (+ 108us) server_negotiation.cc:407] Connection header received
0925 05:09:53.100837 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:53.100838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:53.100848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:53.100899 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:53.101012 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.101535 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.102572 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.102842 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.102906 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:53.103055 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:53.103084 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:53.103137 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:53.103181 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 05:09:53.103200 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 05:09:53.104079 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:53.100616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.14.33.62:37347)
0925 05:09:53.100709 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:53.100711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:53.101013 (+ 302us) server_negotiation.cc:407] Connection header received
0925 05:09:53.101055 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:53.101056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:53.101123 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:53.101206 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:53.101349 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.102083 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.103101 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.103427 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.103487 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:53.103682 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:53.103800 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:53.103851 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:53.103927 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 05:09:53.103960 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":51,"threads_started":1}
I20260925 05:09:53.104563 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:09:53.100923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.14.33.60:39621)
0925 05:09:53.100958 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 05:09:53.100960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 05:09:53.101695 (+ 735us) server_negotiation.cc:407] Connection header received
0925 05:09:53.101767 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:09:53.101768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:09:53.101795 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:09:53.101846 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:09:53.101991 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.102697 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.103780 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:09:53.104061 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:09:53.104093 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:09:53.104323 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:09:53.104351 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:09:53.104405 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 05:09:53.104503 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 05:09:53.104518 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 05:09:53.118120 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4568
I20260925 05:09:53.123292 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4643
I20260925 05:09:53.128178 14468 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task85DeA_/build/release/bin/kudu with pid 4719
2026-09-25T05:09:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2750 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5456 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263553 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST